2024-09-16T01:59:37.990 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T01:59:38.009 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T01:59:38.100 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743 branch: main description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905743' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_object_clean_region_max_num_intervals: 1000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.1 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - slow request - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOto7MfRAfS9JbCWm9JLp5uccsJFf/ne+FZTN65vjqA6GnS05+xcig1k8Rtqxoot0Ngs7Eb7h22jT60ePNPGcEs= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGa2xV9hiF6j9JMIRLjLhG2E2W8r4Lui6MMuyYZ/r0gDu0Sub0NuLmR0nWxrKfnxf45A65YdgCG6REwWz5Vg98= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVExJro8+FA2WStaHpJpkExYYGfjEYa5dqrRETgYswK3+cbtn6KB6kCY9OVdOZhh71ouIVzlVdAtCtLQ8aRGfw= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFQAQe1MhX6xJ+w/15Zm1YkoOIcxz4zHSnH6VP+WKiFgybZyFk1Y4/r0i+ZSMDr3ANOj1H9P11goliuLZPWgxU= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: bluestore_new_sharding: random chance_bluestore_reshard: 1 chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 min_in: 8 timeout: 1200 - rados: clients: - client.0 ec_pool: true erasure_code_profile: c: 2 crush-failure-domain: osd k: 4 m: 3 name: shecprofile plugin: shec objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 400 write_append_excl: false teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T01:59:38.100 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-16T01:59:38.100 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-16T01:59:38.101 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T01:59:38.102 INFO:teuthology.task.internal:Checking packages... 2024-09-16T01:59:38.124 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-16T01:59:38.124 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T01:59:38.124 INFO:teuthology.packaging:ref: None 2024-09-16T01:59:38.124 INFO:teuthology.packaging:tag: None 2024-09-16T01:59:38.124 INFO:teuthology.packaging:branch: main 2024-09-16T01:59:38.125 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T01:59:38.125 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-16T01:59:38.366 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4937-g74984932-1jammy 2024-09-16T01:59:38.368 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T01:59:38.374 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T01:59:38.374 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T01:59:38.403 INFO:teuthology.task.internal:Saving configuration 2024-09-16T01:59:38.425 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T01:59:38.450 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T01:59:38.474 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:53:12.682766', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDl1xoD5HroeZfI8+z9lJHmy1+baUlZQsk9PH+qZz8UJ'} 2024-09-16T01:59:38.498 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:53:12.685416', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL5EujIut/WpjKTn/zzPJc5+mXLQuDjoSMaJWOUfFGMC'} 2024-09-16T01:59:38.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:53:12.684386', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK/SohS9p0BoANcYuRluLAWTL/V0hkakz4/YI8yk/3AG'} 2024-09-16T01:59:38.545 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 01:53:12.686422', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPx2HYO7kHwvs43sGwa3GK5u0DXEA0el6Vm/CuMvw+B+'} 2024-09-16T01:59:38.546 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T01:59:38.552 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-16T01:59:38.552 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-16T01:59:38.552 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-16T01:59:38.552 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-16T01:59:38.552 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T01:59:38.704 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f42620aee60>, signals=[15]) 2024-09-16T01:59:38.704 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T01:59:38.711 INFO:teuthology.task.internal:Opening connections... 2024-09-16T01:59:38.711 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T01:59:38.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:59:38.805 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T01:59:38.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:59:38.885 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T01:59:38.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:59:38.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T01:59:38.963 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T01:59:39.042 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T01:59:39.049 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-09-16T01:59:39.056 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-09-16T01:59:39.056 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-09-16T01:59:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Ubuntu" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="22.04" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_CODENAME=jammy 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:ID=ubuntu 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE=debian 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T01:59:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:UBUNTU_CODENAME=jammy 2024-09-16T01:59:39.106 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-09-16T01:59:39.123 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-09-16T01:59:39.129 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-09-16T01:59:39.129 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-09-16T01:59:39.176 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T01:59:39.176 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="Ubuntu" 2024-09-16T01:59:39.176 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="22.04" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_CODENAME=jammy 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:ID=ubuntu 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE=debian 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T01:59:39.177 INFO:teuthology.orchestra.run.smithi118.stdout:UBUNTU_CODENAME=jammy 2024-09-16T01:59:39.177 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-09-16T01:59:39.198 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-09-16T01:59:39.204 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-09-16T01:59:39.205 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-09-16T01:59:39.250 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T01:59:39.250 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="Ubuntu" 2024-09-16T01:59:39.250 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="22.04" 2024-09-16T01:59:39.250 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T01:59:39.250 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_CODENAME=jammy 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:ID=ubuntu 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE=debian 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T01:59:39.251 INFO:teuthology.orchestra.run.smithi177.stdout:UBUNTU_CODENAME=jammy 2024-09-16T01:59:39.251 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-09-16T01:59:39.264 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-09-16T01:59:39.271 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-09-16T01:59:39.271 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-09-16T01:59:39.317 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Ubuntu" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="22.04" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_CODENAME=jammy 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:ID=ubuntu 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE=debian 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T01:59:39.318 INFO:teuthology.orchestra.run.smithi195.stdout:UBUNTU_CODENAME=jammy 2024-09-16T01:59:39.319 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-09-16T01:59:39.339 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T01:59:39.348 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T01:59:39.399 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T01:59:39.399 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:59:39.402 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:59:39.404 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:59:39.407 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-09-16T01:59:39.412 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T01:59:39.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T01:59:39.437 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:59:39.451 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:59:39.454 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:59:39.457 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-09-16T01:59:39.481 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T01:59:40.209 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T01:59:40.226 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T01:59:40.227 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T01:59:40.227 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:59:40.228 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:59:40.228 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:59:40.229 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T01:59:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:59:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:59:40.235 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-16T01:59:40.235 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-16T01:59:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:59:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T01:59:40.237 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-09-16T01:59:40.237 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-09-16T01:59:40.281 INFO:teuthology.orchestra.run.smithi118.stdout:5.15.0-56-generic 2024-09-16T01:59:40.281 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.15.0-56-generic 2024-09-16T01:59:40.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-09-16T01:59:40.283 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-56-generic 2024-09-16T01:59:40.283 INFO:teuthology.orchestra.run.smithi195.stdout:5.15.0-56-generic 2024-09-16T01:59:40.284 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.15.0-56-generic 2024-09-16T01:59:40.284 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-09-16T01:59:40.284 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.15.0-56-generic 2024-09-16T01:59:40.284 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2024-09-16T01:59:40.286 INFO:teuthology.orchestra.run.smithi003.stdout:5.15.0-56-generic 2024-09-16T01:59:40.287 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.15.0-56-generic 2024-09-16T01:59:40.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get clean 2024-09-16T01:59:40.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-16T01:59:40.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2024-09-16T01:59:40.434 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-16T01:59:40.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-09-16T01:59:40.557 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T01:59:40.588 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T01:59:40.592 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T01:59:40.597 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T01:59:40.597 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T01:59:40.659 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T01:59:40.664 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T01:59:40.692 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T01:59:40.692 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T01:59:40.704 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T01:59:40.746 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T01:59:40.758 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T01:59:40.760 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T01:59:40.789 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T01:59:40.806 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T01:59:40.842 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T01:59:40.864 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T01:59:40.868 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T01:59:40.871 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T01:59:40.871 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T01:59:40.887 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T01:59:40.893 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T01:59:40.893 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T01:59:40.899 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T01:59:40.899 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T01:59:40.905 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T01:59:40.910 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T01:59:40.913 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T01:59:40.915 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T01:59:40.915 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T01:59:40.915 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T01:59:40.920 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T01:59:40.922 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T01:59:40.924 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T01:59:40.928 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T01:59:40.928 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T01:59:40.947 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T01:59:40.947 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T01:59:40.948 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T01:59:40.948 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T01:59:40.949 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T01:59:40.950 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T01:59:40.950 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T01:59:40.952 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T01:59:40.953 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T01:59:40.958 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T01:59:40.964 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T01:59:40.966 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T01:59:40.966 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T01:59:40.966 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T01:59:40.979 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T01:59:40.979 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T01:59:40.979 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T01:59:40.982 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T01:59:40.982 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T01:59:40.988 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T01:59:40.992 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T01:59:40.993 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T01:59:40.993 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T01:59:40.994 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T01:59:40.997 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T01:59:40.999 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T01:59:41.009 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T01:59:41.017 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T01:59:41.022 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T01:59:41.022 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T01:59:41.023 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T01:59:41.051 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T01:59:41.056 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T01:59:41.057 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T01:59:41.068 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T01:59:41.075 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T01:59:41.078 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T01:59:41.079 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T01:59:41.079 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T01:59:41.080 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T01:59:41.080 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T01:59:41.090 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T01:59:41.091 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T01:59:41.092 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T01:59:41.102 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T01:59:41.107 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T01:59:41.107 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T01:59:41.107 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T01:59:41.108 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T01:59:41.108 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T01:59:41.112 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T01:59:41.135 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T01:59:41.141 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T01:59:41.241 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T01:59:41.250 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T01:59:41.281 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T01:59:41.559 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T01:59:41.568 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T01:59:41.568 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T01:59:41.584 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T01:59:41.593 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T01:59:41.594 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T01:59:41.594 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T01:59:41.595 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T01:59:41.598 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T01:59:41.603 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T01:59:41.610 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T01:59:41.611 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T01:59:41.611 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T01:59:41.618 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T01:59:41.620 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T01:59:41.634 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T01:59:41.648 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T01:59:41.649 INFO:teuthology.orchestra.run.smithi003.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T01:59:41.681 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T01:59:41.688 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T01:59:41.690 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T01:59:41.691 INFO:teuthology.orchestra.run.smithi195.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T01:59:41.694 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T01:59:41.695 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T01:59:41.718 INFO:teuthology.orchestra.run.smithi003.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T01:59:41.719 INFO:teuthology.orchestra.run.smithi003.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T01:59:41.730 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T01:59:41.734 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T01:59:41.735 INFO:teuthology.orchestra.run.smithi003.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T01:59:41.735 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T01:59:41.735 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T01:59:41.735 INFO:teuthology.orchestra.run.smithi003.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T01:59:41.736 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T01:59:41.746 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T01:59:41.747 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T01:59:41.753 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T01:59:41.754 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T01:59:41.755 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T01:59:41.755 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T01:59:41.756 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T01:59:41.762 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T01:59:41.765 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T01:59:41.765 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T01:59:41.766 INFO:teuthology.orchestra.run.smithi003.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T01:59:41.767 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T01:59:41.767 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T01:59:41.770 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T01:59:41.774 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T01:59:41.774 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T01:59:41.775 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T01:59:41.786 INFO:teuthology.orchestra.run.smithi003.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T01:59:41.792 INFO:teuthology.orchestra.run.smithi003.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T01:59:41.792 INFO:teuthology.orchestra.run.smithi003.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T01:59:41.793 INFO:teuthology.orchestra.run.smithi003.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T01:59:41.806 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T01:59:41.806 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T01:59:41.807 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T01:59:41.814 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T01:59:41.814 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T01:59:41.835 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T01:59:41.838 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T01:59:41.838 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T01:59:41.838 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T01:59:41.839 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T01:59:41.840 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T01:59:41.844 INFO:teuthology.orchestra.run.smithi003.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T01:59:41.845 INFO:teuthology.orchestra.run.smithi003.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T01:59:41.845 INFO:teuthology.orchestra.run.smithi003.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T01:59:41.847 INFO:teuthology.orchestra.run.smithi003.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T01:59:41.857 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T01:59:41.859 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T01:59:41.860 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T01:59:41.861 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T01:59:41.862 INFO:teuthology.orchestra.run.smithi195.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T01:59:41.876 INFO:teuthology.orchestra.run.smithi003.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T01:59:41.877 INFO:teuthology.orchestra.run.smithi003.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T01:59:41.877 INFO:teuthology.orchestra.run.smithi003.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T01:59:41.877 INFO:teuthology.orchestra.run.smithi003.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T01:59:41.878 INFO:teuthology.orchestra.run.smithi003.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T01:59:41.887 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T01:59:41.895 INFO:teuthology.orchestra.run.smithi195.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T01:59:41.896 INFO:teuthology.orchestra.run.smithi195.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T01:59:41.897 INFO:teuthology.orchestra.run.smithi195.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T01:59:41.926 INFO:teuthology.orchestra.run.smithi003.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T01:59:47.236 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 16.0 MB in 2s (7,333 kB/s) 2024-09-16T01:59:47.246 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 16.0 MB in 2s (7,248 kB/s) 2024-09-16T01:59:47.663 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 16.0 MB in 2s (7,253 kB/s) 2024-09-16T01:59:48.035 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 16.0 MB in 3s (5,718 kB/s) 2024-09-16T01:59:48.622 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T01:59:48.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T01:59:48.654 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T01:59:48.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T01:59:48.701 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T01:59:48.737 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T01:59:48.914 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-16T01:59:48.914 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-16T01:59:48.949 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-16T01:59:48.950 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-16T01:59:49.089 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T01:59:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T01:59:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T01:59:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T01:59:49.089 INFO:teuthology.orchestra.run.smithi177.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T01:59:49.090 INFO:teuthology.orchestra.run.smithi177.stdout: python2.7-minimal 2024-09-16T01:59:49.090 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T01:59:49.091 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-09-16T01:59:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T01:59:49.091 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.092 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-09-16T01:59:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T01:59:49.093 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-09-16T01:59:49.093 INFO:teuthology.orchestra.run.smithi177.stdout: thermald 2024-09-16T01:59:49.139 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-09-16T01:59:49.139 INFO:teuthology.orchestra.run.smithi177.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T01:59:49.140 INFO:teuthology.orchestra.run.smithi177.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.141 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages will be upgraded: 2024-09-16T01:59:49.142 INFO:teuthology.orchestra.run.smithi177.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T01:59:49.144 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T01:59:49.144 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T01:59:49.144 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T01:59:49.145 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T01:59:49.145 INFO:teuthology.orchestra.run.smithi118.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T01:59:49.145 INFO:teuthology.orchestra.run.smithi118.stdout: python2.7-minimal 2024-09-16T01:59:49.145 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T01:59:49.147 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-09-16T01:59:49.147 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.147 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T01:59:49.147 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.149 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-09-16T01:59:49.149 INFO:teuthology.orchestra.run.smithi118.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T01:59:49.149 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-09-16T01:59:49.149 INFO:teuthology.orchestra.run.smithi118.stdout: thermald 2024-09-16T01:59:49.190 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T01:59:49.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T01:59:49.209 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-09-16T01:59:49.209 INFO:teuthology.orchestra.run.smithi118.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.209 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T01:59:49.209 INFO:teuthology.orchestra.run.smithi118.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.211 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages will be upgraded: 2024-09-16T01:59:49.211 INFO:teuthology.orchestra.run.smithi118.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T01:59:49.262 INFO:teuthology.orchestra.run.smithi118.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T01:59:49.262 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 113 MB of archives. 2024-09-16T01:59:49.262 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T01:59:49.262 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T01:59:49.263 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T01:59:49.323 INFO:teuthology.orchestra.run.smithi177.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T01:59:49.323 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 113 MB of archives. 2024-09-16T01:59:49.323 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T01:59:49.323 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T01:59:49.458 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-09-16T01:59:49.459 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-09-16T01:59:49.473 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T01:59:49.491 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T01:59:49.553 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T01:59:49.635 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T01:59:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T01:59:49.635 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T01:59:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T01:59:49.636 INFO:teuthology.orchestra.run.smithi003.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T01:59:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: python2.7-minimal 2024-09-16T01:59:49.637 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T01:59:49.637 INFO:teuthology.orchestra.run.smithi003.stdout:The following additional packages will be installed: 2024-09-16T01:59:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.637 INFO:teuthology.orchestra.run.smithi003.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T01:59:49.638 INFO:teuthology.orchestra.run.smithi003.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.639 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2024-09-16T01:59:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T01:59:49.639 INFO:teuthology.orchestra.run.smithi003.stdout:Recommended packages: 2024-09-16T01:59:49.639 INFO:teuthology.orchestra.run.smithi003.stdout: thermald 2024-09-16T01:59:49.688 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2024-09-16T01:59:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T01:59:49.689 INFO:teuthology.orchestra.run.smithi003.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.690 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages will be upgraded: 2024-09-16T01:59:49.691 INFO:teuthology.orchestra.run.smithi003.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T01:59:49.701 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T01:59:49.741 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2024-09-16T01:59:49.741 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2024-09-16T01:59:49.795 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T01:59:49.871 INFO:teuthology.orchestra.run.smithi003.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T01:59:49.871 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 113 MB of archives. 2024-09-16T01:59:49.871 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T01:59:49.871 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout: python2.7-minimal 2024-09-16T01:59:49.912 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T01:59:49.913 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2024-09-16T01:59:49.913 INFO:teuthology.orchestra.run.smithi195.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.913 INFO:teuthology.orchestra.run.smithi195.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T01:59:49.914 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.915 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2024-09-16T01:59:49.915 INFO:teuthology.orchestra.run.smithi195.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T01:59:49.915 INFO:teuthology.orchestra.run.smithi195.stdout:Recommended packages: 2024-09-16T01:59:49.916 INFO:teuthology.orchestra.run.smithi195.stdout: thermald 2024-09-16T01:59:49.975 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2024-09-16T01:59:49.975 INFO:teuthology.orchestra.run.smithi195.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T01:59:49.975 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T01:59:49.975 INFO:teuthology.orchestra.run.smithi195.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T01:59:49.976 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages will be upgraded: 2024-09-16T01:59:49.977 INFO:teuthology.orchestra.run.smithi195.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T01:59:50.028 INFO:teuthology.orchestra.run.smithi195.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T01:59:50.028 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 113 MB of archives. 2024-09-16T01:59:50.028 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T01:59:50.028 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T01:59:50.449 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T01:59:50.537 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T01:59:50.761 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T01:59:50.875 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T01:59:50.875 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T01:59:50.875 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T01:59:50.999 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T01:59:51.073 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T01:59:51.113 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T01:59:51.277 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T01:59:51.326 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T01:59:51.327 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T01:59:51.328 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T01:59:51.449 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T01:59:51.469 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T01:59:51.483 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 113 MB in 2s (61.1 MB/s) 2024-09-16T01:59:51.634 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T01:59:51.641 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T01:59:51.863 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 113 MB in 1s (76.3 MB/s) 2024-09-16T01:59:52.036 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T01:59:53.348 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T01:59:53.349 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T01:59:53.349 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T01:59:53.724 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T01:59:53.804 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T01:59:53.842 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T01:59:53.842 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T01:59:53.842 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T01:59:54.012 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T01:59:54.016 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:54.070 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T01:59:54.074 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:54.142 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:54.171 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:54.194 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T01:59:54.220 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 113 MB in 5s (24.3 MB/s) 2024-09-16T01:59:54.278 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T01:59:54.404 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T01:59:54.715 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2024-09-16T01:59:54.883 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T01:59:56.226 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T01:59:56.247 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:56.339 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:56.425 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T01:59:56.430 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:56.509 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:56.821 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T01:59:56.842 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:56.879 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:56.930 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T01:59:56.951 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:56.989 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T01:59:56.993 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:57.038 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:57.315 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:57.608 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T01:59:57.628 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:57.699 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:58.835 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T01:59:58.856 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:58.945 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:59.349 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T01:59:59.371 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:59.446 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T01:59:59.462 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:59.467 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:59.504 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T01:59:59.911 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T01:59:59.931 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T01:59:59.977 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:03.279 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:03.398 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:03.739 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:03.843 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:04.129 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:00:04.151 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:00:04.196 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:05.007 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:05.135 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:05.425 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:05.539 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:05.842 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:00:05.864 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:00:05.918 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:05.933 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:06.029 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:06.337 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:06.482 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:06.568 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:06.685 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:06.768 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:00:06.779 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:00:06.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:07.035 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:07.146 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:07.457 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T02:00:07.468 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T02:00:07.507 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:10.090 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:00:10.129 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:00:10.173 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:12.002 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:00:12.040 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:00:12.085 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:12.680 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:00:12.717 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:00:12.754 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:12.848 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:12.977 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:13.110 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T02:00:13.147 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T02:00:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:13.328 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:13.448 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:13.574 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:00:13.677 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:14.760 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:14.883 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:15.002 INFO:teuthology.orchestra.run.smithi195.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:15.002 INFO:teuthology.orchestra.run.smithi195.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:15.002 INFO:teuthology.orchestra.run.smithi195.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:15.002 INFO:teuthology.orchestra.run.smithi195.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:15.155 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:15.275 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:15.377 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:16.347 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:16.492 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:16.507 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:00:16.635 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:16.815 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:16.819 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:16.947 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:17.074 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:00:17.078 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T02:00:17.201 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:17.212 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T02:00:17.597 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T02:00:17.716 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:17.817 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T02:00:17.928 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:17.980 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:17.980 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:17.981 INFO:teuthology.orchestra.run.smithi118.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:17.981 INFO:teuthology.orchestra.run.smithi118.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:18.157 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:18.182 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:00:18.309 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:00:18.435 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:18.496 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:00:18.496 INFO:teuthology.orchestra.run.smithi195.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:18.590 INFO:teuthology.orchestra.run.smithi177.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:18.746 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:19.276 INFO:teuthology.orchestra.run.smithi003.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:19.276 INFO:teuthology.orchestra.run.smithi003.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:19.276 INFO:teuthology.orchestra.run.smithi003.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:19.277 INFO:teuthology.orchestra.run.smithi003.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:19.472 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:19.558 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:20.163 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:20.906 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:20.951 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:00:21.069 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:00:21.195 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:21.264 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:00:21.264 INFO:teuthology.orchestra.run.smithi118.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:21.873 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:00:22.474 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:00:22.609 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:22.668 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:00:22.668 INFO:teuthology.orchestra.run.smithi177.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:22.783 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T02:00:22.900 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T02:00:23.018 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T02:00:23.079 INFO:teuthology.orchestra.run.smithi003.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T02:00:23.080 INFO:teuthology.orchestra.run.smithi003.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:32.258 INFO:teuthology.orchestra.run.smithi195.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:00:32.258 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:00:32.267 INFO:teuthology.orchestra.run.smithi195.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:32.291 INFO:teuthology.orchestra.run.smithi195.stdout:Generating grub configuration file ... 2024-09-16T02:00:32.585 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:32.608 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:32.894 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:32.900 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:32.961 INFO:teuthology.orchestra.run.smithi195.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:32.966 INFO:teuthology.orchestra.run.smithi195.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:33.126 INFO:teuthology.orchestra.run.smithi195.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:33.126 INFO:teuthology.orchestra.run.smithi195.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:33.126 INFO:teuthology.orchestra.run.smithi195.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:33.172 INFO:teuthology.orchestra.run.smithi195.stdout:done 2024-09-16T02:00:33.653 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 21 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta 2024-09-16T02:00:33.673 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:33.674 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2024-09-16T02:00:33.674 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:00:33.674 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:00:33.674 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:00:33.674 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-16T02:00:33.675 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:00:33.675 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get clean 2024-09-16T02:00:33.850 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2024-09-16T02:00:33.975 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:00:33.976 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:00:33.992 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:00:34.090 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:00:35.412 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T02:00:35.431 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:00:35.493 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T02:00:35.704 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2024-09-16T02:00:35.704 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2024-09-16T02:00:35.894 INFO:teuthology.orchestra.run.smithi195.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:00:35.894 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:00:35.894 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:00:35.894 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:00:35.894 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:00:35.895 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:00:35.895 INFO:teuthology.orchestra.run.smithi195.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:00:35.895 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:00:35.908 INFO:teuthology.orchestra.run.smithi118.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:00:35.908 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:00:35.936 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:00:35.938 DEBUG:teuthology.orchestra.run.smithi195:> dpkg -s linux-image-generic 2024-09-16T02:00:35.938 INFO:teuthology.orchestra.run.smithi118.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:35.953 INFO:teuthology.orchestra.run.smithi118.stdout:Generating grub configuration file ... 2024-09-16T02:00:35.953 INFO:teuthology.orchestra.run.smithi195.stdout:Package: linux-image-generic 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Status: install ok installed 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Priority: optional 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Section: kernel 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Installed-Size: 21 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: amd64 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Source: linux-meta 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:35.954 INFO:teuthology.orchestra.run.smithi195.stdout:Recommends: thermald 2024-09-16T02:00:35.955 INFO:teuthology.orchestra.run.smithi195.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:35.955 INFO:teuthology.orchestra.run.smithi195.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:35.955 INFO:teuthology.orchestra.run.smithi195.stdout: available. 2024-09-16T02:00:35.955 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-09-16T02:00:36.001 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.nWuaw6lk2S 2024-09-16T02:00:36.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub/grub.cfg /tmp/tmp.nWuaw6lk2S 2024-09-16T02:00:36.056 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.nWuaw6lk2S 2024-09-16T02:00:36.168 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.nWuaw6lk2S is 10KB 2024-09-16T02:00:36.180 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.nWuaw6lk2S 2024-09-16T02:00:36.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:36.227 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:36.240 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:00:36.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:00:36.240 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:36.242 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:36.308 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:00:36.308 DEBUG:teuthology.orchestra.run.smithi195:> sudo update-grub 2024-09-16T02:00:36.525 INFO:teuthology.orchestra.run.smithi177.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:00:36.525 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:00:36.531 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:36.536 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:36.542 INFO:teuthology.orchestra.run.smithi177.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:36.555 INFO:teuthology.orchestra.run.smithi177.stdout:Generating grub configuration file ... 2024-09-16T02:00:36.605 INFO:teuthology.orchestra.run.smithi118.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:36.611 INFO:teuthology.orchestra.run.smithi118.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:36.750 INFO:teuthology.orchestra.run.smithi118.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:36.750 INFO:teuthology.orchestra.run.smithi118.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:36.750 INFO:teuthology.orchestra.run.smithi118.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:36.785 INFO:teuthology.orchestra.run.smithi118.stdout:done 2024-09-16T02:00:36.852 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:36.872 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:37.164 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:37.169 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:37.240 INFO:teuthology.orchestra.run.smithi177.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:37.245 INFO:teuthology.orchestra.run.smithi177.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:37.284 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 21 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:37.309 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-09-16T02:00:37.310 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:37.310 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:37.310 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-09-16T02:00:37.310 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:00:37.310 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:00:37.310 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:00:37.310 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-16T02:00:37.310 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:00:37.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get clean 2024-09-16T02:00:37.393 INFO:teuthology.orchestra.run.smithi177.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:37.393 INFO:teuthology.orchestra.run.smithi177.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:37.393 INFO:teuthology.orchestra.run.smithi177.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:37.428 INFO:teuthology.orchestra.run.smithi177.stdout:done 2024-09-16T02:00:37.496 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:00:37.497 INFO:teuthology.orchestra.run.smithi195.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:37.499 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-09-16T02:00:37.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-16T02:00:37.645 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:00:37.645 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:00:37.654 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:00:37.671 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:00:37.689 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:37.695 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:37.700 INFO:teuthology.orchestra.run.smithi003.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T02:00:37.701 INFO:teuthology.orchestra.run.smithi003.stdout:Sourcing file `/etc/default/grub' 2024-09-16T02:00:37.718 INFO:teuthology.orchestra.run.smithi003.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:37.740 INFO:teuthology.orchestra.run.smithi003.stdout:Generating grub configuration file ... 2024-09-16T02:00:37.921 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:37.927 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:37.931 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-09-16T02:00:37.957 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-09-16T02:00:37.957 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:37.958 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:37.959 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-09-16T02:00:37.959 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:37.959 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:37.959 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-09-16T02:00:37.960 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:00:37.960 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:00:37.960 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:00:37.960 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-16T02:00:37.960 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:00:37.961 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get clean 2024-09-16T02:00:37.961 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:37.968 INFO:teuthology.orchestra.run.smithi195.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:37.974 INFO:teuthology.orchestra.run.smithi195.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:37.982 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:38.098 INFO:teuthology.orchestra.run.smithi195.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:38.098 INFO:teuthology.orchestra.run.smithi195.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:38.098 INFO:teuthology.orchestra.run.smithi195.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:38.123 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-09-16T02:00:38.131 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-09-16T02:00:38.135 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-16T02:00:38.258 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:00:38.258 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:00:38.268 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:00:38.285 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:00:38.310 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:38.316 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:38.384 INFO:teuthology.orchestra.run.smithi003.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:38.390 INFO:teuthology.orchestra.run.smithi003.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:38.558 INFO:teuthology.orchestra.run.smithi003.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:38.559 INFO:teuthology.orchestra.run.smithi003.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:38.559 INFO:teuthology.orchestra.run.smithi003.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:38.596 INFO:teuthology.orchestra.run.smithi003.stdout:done 2024-09-16T02:00:38.994 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T02:00:39.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:00:39.066 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T02:00:39.082 DEBUG:teuthology.orchestra.run.smithi003:> dpkg -s linux-image-generic 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Package: linux-image-generic 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Status: install ok installed 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Priority: optional 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Section: kernel 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Installed-Size: 21 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:39.103 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: amd64 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Source: linux-meta 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Recommends: thermald 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:39.104 INFO:teuthology.orchestra.run.smithi003.stdout: available. 2024-09-16T02:00:39.104 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T02:00:39.104 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T02:00:39.104 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T02:00:39.105 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T02:00:39.105 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T02:00:39.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get clean 2024-09-16T02:00:39.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-09-16T02:00:39.251 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-16T02:00:39.252 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-16T02:00:39.362 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:00:39.362 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:00:39.372 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:00:39.389 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:00:39.419 INFO:teuthology.orchestra.run.smithi118.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:00:39.420 INFO:teuthology.orchestra.run.smithi118.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:00:39.421 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:00:39.460 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:00:39.462 DEBUG:teuthology.orchestra.run.smithi118:> dpkg -s linux-image-generic 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Package: linux-image-generic 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Status: install ok installed 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Priority: optional 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Section: kernel 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Installed-Size: 21 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: amd64 2024-09-16T02:00:39.515 INFO:teuthology.orchestra.run.smithi118.stdout:Source: linux-meta 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout:Recommends: thermald 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:39.516 INFO:teuthology.orchestra.run.smithi118.stdout: available. 2024-09-16T02:00:39.516 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-16T02:00:39.563 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.USWyEiTs3H 2024-09-16T02:00:39.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub/grub.cfg /tmp/tmp.USWyEiTs3H 2024-09-16T02:00:39.623 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.USWyEiTs3H 2024-09-16T02:00:39.649 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T02:00:39.668 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:00:39.721 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T02:00:39.746 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.USWyEiTs3H is 10KB 2024-09-16T02:00:39.758 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.USWyEiTs3H 2024-09-16T02:00:39.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:39.818 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:00:39.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:00:39.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:39.886 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:00:39.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo update-grub 2024-09-16T02:00:39.913 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-16T02:00:39.913 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-16T02:00:40.080 INFO:teuthology.orchestra.run.smithi177.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:00:40.080 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:00:40.080 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:00:40.080 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:00:40.080 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:00:40.081 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:00:40.081 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:00:40.081 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:00:40.121 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:00:40.123 DEBUG:teuthology.orchestra.run.smithi177:> dpkg -s linux-image-generic 2024-09-16T02:00:40.179 INFO:teuthology.orchestra.run.smithi177.stdout:Package: linux-image-generic 2024-09-16T02:00:40.179 INFO:teuthology.orchestra.run.smithi177.stdout:Status: install ok installed 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Priority: optional 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Section: kernel 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Installed-Size: 21 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: amd64 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Source: linux-meta 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Recommends: thermald 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:40.180 INFO:teuthology.orchestra.run.smithi177.stdout: available. 2024-09-16T02:00:40.181 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-16T02:00:40.224 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.v22GNHAKWp 2024-09-16T02:00:40.224 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub/grub.cfg /tmp/tmp.v22GNHAKWp 2024-09-16T02:00:40.281 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.v22GNHAKWp 2024-09-16T02:00:40.394 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.v22GNHAKWp is 10KB 2024-09-16T02:00:40.406 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.v22GNHAKWp 2024-09-16T02:00:40.411 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:40.468 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:00:40.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:00:40.468 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:40.540 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:00:40.540 DEBUG:teuthology.orchestra.run.smithi177:> sudo update-grub 2024-09-16T02:00:40.809 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T02:00:40.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T02:00:40.892 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T02:00:41.120 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-09-16T02:00:41.120 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-09-16T02:00:41.320 INFO:teuthology.orchestra.run.smithi003.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T02:00:41.320 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:00:41.320 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:00:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:00:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:00:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:00:41.321 INFO:teuthology.orchestra.run.smithi003.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:00:41.321 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:00:41.364 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T02:00:41.367 DEBUG:teuthology.orchestra.run.smithi003:> dpkg -s linux-image-generic 2024-09-16T02:00:41.382 INFO:teuthology.orchestra.run.smithi003.stdout:Package: linux-image-generic 2024-09-16T02:00:41.382 INFO:teuthology.orchestra.run.smithi003.stdout:Status: install ok installed 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Priority: optional 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Section: kernel 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Installed-Size: 21 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: amd64 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Source: linux-meta 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Version: 5.15.0.121.121 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Recommends: thermald 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout:Description: Generic Linux kernel image 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout: This package will always depend on the latest generic kernel image 2024-09-16T02:00:41.383 INFO:teuthology.orchestra.run.smithi003.stdout: available. 2024-09-16T02:00:41.384 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-09-16T02:00:41.427 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.y9wRITuhML 2024-09-16T02:00:41.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub/grub.cfg /tmp/tmp.y9wRITuhML 2024-09-16T02:00:41.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.y9wRITuhML 2024-09-16T02:00:41.596 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.y9wRITuhML is 10KB 2024-09-16T02:00:41.610 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.y9wRITuhML 2024-09-16T02:00:41.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:41.639 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:00:41.640 INFO:teuthology.orchestra.run.smithi118.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:41.643 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-09-16T02:00:41.668 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:00:41.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T02:00:41.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T02:00:41.745 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T02:00:41.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo update-grub 2024-09-16T02:00:41.792 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:00:41.793 INFO:teuthology.orchestra.run.smithi177.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:41.798 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-09-16T02:00:41.818 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:41.824 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:41.991 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:41.997 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:42.075 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:42.079 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:42.118 INFO:teuthology.orchestra.run.smithi118.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:42.123 INFO:teuthology.orchestra.run.smithi118.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:42.233 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:42.238 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:42.279 INFO:teuthology.orchestra.run.smithi177.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:42.282 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:42.282 INFO:teuthology.orchestra.run.smithi118.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:42.282 INFO:teuthology.orchestra.run.smithi118.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:42.285 INFO:teuthology.orchestra.run.smithi177.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:42.315 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-09-16T02:00:42.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-09-16T02:00:42.406 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:42.406 INFO:teuthology.orchestra.run.smithi177.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:42.406 INFO:teuthology.orchestra.run.smithi177.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:42.435 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-09-16T02:00:42.448 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-09-16T02:00:42.999 INFO:teuthology.orchestra.run.smithi003.stderr:Sourcing file `/etc/default/grub' 2024-09-16T02:00:43.001 INFO:teuthology.orchestra.run.smithi003.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T02:00:43.002 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-09-16T02:00:43.186 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T02:00:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T02:00:43.470 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T02:00:43.476 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T02:00:43.519 INFO:teuthology.orchestra.run.smithi003.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T02:00:43.524 INFO:teuthology.orchestra.run.smithi003.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T02:00:43.645 INFO:teuthology.orchestra.run.smithi003.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T02:00:43.645 INFO:teuthology.orchestra.run.smithi003.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T02:00:43.645 INFO:teuthology.orchestra.run.smithi003.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T02:00:43.662 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-09-16T02:00:43.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-09-16T02:01:08.157 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-16T02:01:08.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:01:08.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:12.325 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-16T02:01:12.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:01:12.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:12.450 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-16T02:01:12.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:01:12.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:13.667 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T02:01:13.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:01:13.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:26.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-09-16T02:01:30.685 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-16T02:01:30.941 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-16T02:01:32.225 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-09-16T02:01:35.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:01:35.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:39.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:01:39.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:39.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:01:39.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:41.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:01:41.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:42.749 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-16T02:01:43.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-16T02:01:44.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-09-16T02:01:53.985 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-09-16T02:01:54.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:01:54.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:55.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:01:55.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:56.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:01:56.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:01:57.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-09-16T02:02:05.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:02:05.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:02:09.053 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-09-16T02:02:12.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:02:12.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:02:13.441 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-09-16T02:02:24.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:02:24.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:02:24.307 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-09-16T02:02:24.923 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:02:24.923 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-121-generic"... 2024-09-16T02:02:24.923 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-09-16T02:02:24.970 INFO:teuthology.orchestra.run.smithi195.stdout:5.15.0-121-generic 2024-09-16T02:02:24.970 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:02:24.970 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:02:24.970 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:02:25.971 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-16T02:02:25.972 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:02:26.025 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-09-16T02:02:26.059 DEBUG:teuthology.parallel:result is None 2024-09-16T02:02:28.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:02:28.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:02:28.930 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-16T02:02:29.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:02:29.794 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-121-generic"... 2024-09-16T02:02:29.794 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-09-16T02:02:29.842 INFO:teuthology.orchestra.run.smithi177.stdout:5.15.0-121-generic 2024-09-16T02:02:29.842 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:02:29.842 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:02:29.843 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:02:30.843 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-16T02:02:30.844 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:02:30.849 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-09-16T02:02:30.913 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-09-16T02:02:30.940 DEBUG:teuthology.parallel:result is None 2024-09-16T02:02:44.773 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-16T02:02:45.388 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:02:45.389 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-16T02:02:45.389 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-09-16T02:02:45.438 INFO:teuthology.orchestra.run.smithi118.stdout:5.15.0-121-generic 2024-09-16T02:02:45.439 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:02:45.439 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:02:45.439 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:02:45.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:02:45.850 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:02:46.104 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-09-16T02:02:46.439 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-16T02:02:46.440 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:02:46.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:02:46.682 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T02:02:46.683 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-09-16T02:02:46.728 INFO:teuthology.orchestra.run.smithi003.stdout:5.15.0-121-generic 2024-09-16T02:02:46.728 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T02:02:46.728 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T02:02:46.728 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T02:02:46.778 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-09-16T02:02:46.808 DEBUG:teuthology.parallel:result is None 2024-09-16T02:02:47.730 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T02:02:47.730 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T02:02:47.810 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-09-16T02:02:47.836 DEBUG:teuthology.parallel:result is None 2024-09-16T02:02:47.837 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T02:02:47.845 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T02:02:47.845 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:02:47.848 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:02:47.850 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:02:47.853 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T02:02:47.860 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T02:02:47.866 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T02:02:47.882 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T02:02:47.882 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:02:47.897 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:02:47.900 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:02:47.903 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T02:02:48.019 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T02:02:48.025 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T02:02:48.025 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:02:48.027 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:02:48.030 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:02:48.032 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T02:02:48.045 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.049 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.052 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.053 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.057 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.058 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.060 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.066 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T02:02:48.067 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T02:02:48.075 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T02:02:48.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:02:48.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:02:48.103 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:02:48.112 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T02:02:48.128 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T02:02:48.138 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T02:02:48.138 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:02:48.165 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:02:48.167 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:02:48.170 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T02:02:48.176 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:02:48.221 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:02:48.269 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:02:48.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:02:48.333 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:02:48.338 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:02:48.387 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:02:48.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:02:48.475 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:02:48.481 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:02:48.531 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:02:48.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:02:48.597 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T02:02:48.602 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T02:02:48.653 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:02:48.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T02:02:48.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-09-16T02:02:48.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-09-16T02:02:48.715 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-09-16T02:02:48.717 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-09-16T02:02:48.793 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T02:02:48.801 INFO:teuthology.task.internal:Starting timer... 2024-09-16T02:02:48.801 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T02:02:48.845 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Excluding smithi003: OS 'ubuntu' does not support SELinux 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Excluding smithi118: OS 'ubuntu' does not support SELinux 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Excluding smithi177: OS 'ubuntu' does not support SELinux 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Excluding smithi195: OS 'ubuntu' does not support SELinux 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T02:02:48.859 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T02:02:48.860 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T02:02:48.860 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T02:02:48.870 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T02:02:48.873 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T02:02:48.999 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T02:02:49.010 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T02:02:49.011 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi118.front.sepia.ceph.com,smithi177.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T02:10:29.012 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-09-16T02:10:29.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:10:29.038 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:10:29.123 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-09-16T02:10:29.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-09-16T02:10:29.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:10:29.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:10:29.282 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-09-16T02:10:29.360 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-09-16T02:10:29.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:10:29.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:10:29.440 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-09-16T02:10:29.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-09-16T02:10:29.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:10:29.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T02:10:29.593 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-09-16T02:10:29.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-09-16T02:10:29.670 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T02:10:29.679 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T02:10:29.680 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T02:10:29.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:10:29.682 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T02:10:29.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:10:29.685 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T02:10:29.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:10:29.687 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T02:10:29.687 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:10:29.710 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Command line: ntpd -gq 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: ---------------------------------------------------- 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: corporation. Support and training for ntp-4 are 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: available at https://www.nwtime.org/support 2024-09-16T02:10:29.711 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: ---------------------------------------------------- 2024-09-16T02:10:29.712 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: proto: precision = 0.061 usec (-24) 2024-09-16T02:10:29.713 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: basedate set to 2022-02-04 2024-09-16T02:10:29.713 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: gps base set to 2022-02-06 (week 2196) 2024-09-16T02:10:29.713 INFO:teuthology.orchestra.run.smithi003.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.713 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.713 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.714 INFO:teuthology.orchestra.run.smithi003.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.714 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T02:10:29.714 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T02:10:29.714 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T02:10:29.715 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen normally on 3 ens1f0 172.21.15.3:123 2024-09-16T02:10:29.715 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen normally on 4 lo [::1]:123 2024-09-16T02:10:29.715 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15dc%3]:123 2024-09-16T02:10:29.715 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:29 ntpd[18405]: Listening on routing socket on fd #22 for interface updates 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Command line: ntpd -gq 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: ---------------------------------------------------- 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: corporation. Support and training for ntp-4 are 2024-09-16T02:10:29.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: available at https://www.nwtime.org/support 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: ---------------------------------------------------- 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: proto: precision = 0.060 usec (-24) 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: basedate set to 2022-02-04 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: gps base set to 2022-02-06 (week 2196) 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi177.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Command line: ntpd -gq 2024-09-16T02:10:29.718 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: ---------------------------------------------------- 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: corporation. Support and training for ntp-4 are 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: available at https://www.nwtime.org/support 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: ---------------------------------------------------- 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: proto: precision = 0.059 usec (-24) 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: basedate set to 2022-02-04 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: gps base set to 2022-02-06 (week 2196) 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.719 INFO:teuthology.orchestra.run.smithi118.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.720 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.720 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.720 INFO:teuthology.orchestra.run.smithi177.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.720 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.721 INFO:teuthology.orchestra.run.smithi118.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.722 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T02:10:29.722 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T02:10:29.722 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T02:10:29.722 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen normally on 3 enp3s0f1 172.21.15.177:123 2024-09-16T02:10:29.722 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen normally on 4 lo [::1]:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feea:b04b%5]:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:29 ntpd[18486]: Listening on routing socket on fd #22 for interface updates 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen normally on 3 enp3s0f1 172.21.15.118:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen normally on 4 lo [::1]:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d725%5]:123 2024-09-16T02:10:29.723 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:29 ntpd[18484]: Listening on routing socket on fd #22 for interface updates 2024-09-16T02:10:29.739 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Command line: ntpd -gq 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: ---------------------------------------------------- 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: corporation. Support and training for ntp-4 are 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: available at https://www.nwtime.org/support 2024-09-16T02:10:29.740 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: ---------------------------------------------------- 2024-09-16T02:10:29.741 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: proto: precision = 0.061 usec (-24) 2024-09-16T02:10:29.741 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: basedate set to 2022-02-04 2024-09-16T02:10:29.741 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: gps base set to 2022-02-06 (week 2196) 2024-09-16T02:10:29.741 INFO:teuthology.orchestra.run.smithi195.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.742 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T02:10:29.742 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.742 INFO:teuthology.orchestra.run.smithi195.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen normally on 3 enp3s0f1 172.21.15.195:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen normally on 4 lo [::1]:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:8081%5]:123 2024-09-16T02:10:29.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:29 ntpd[18465]: Listening on routing socket on fd #22 for interface updates 2024-09-16T02:10:30.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:30 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:30.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:30 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:30.715 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:30 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:30.715 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:30 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:30.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:30 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:30.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:30 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:31.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:31 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:31.715 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:31.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:31 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:31.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:31.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:31.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:31.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:31 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:32.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:32 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:33 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.717 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:33 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.718 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.718 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.718 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.718 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.718 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:33 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:33.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:33 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:34.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:34.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:34 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:34 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:34.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:34 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:34.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:34.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:34 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:35 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:35 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:35 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:35.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:35 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:36.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:36 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:36.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:36 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:36.715 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:36.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:36 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:36.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.740 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:36.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:36.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:36.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:36 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.711 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:16 Sep 02:10:37 ntpd[18405]: ntpd: time slew -0.000434 s 2024-09-16T02:10:37.712 INFO:teuthology.orchestra.run.smithi003.stdout:ntpd: time slew -0.000434s 2024-09-16T02:10:37.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T02:10:37.713 INFO:teuthology.orchestra.run.smithi003.stderr:16 Sep 02:10:37 ntpd[18405]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T02:10:37.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:37.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:37 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.716 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:37.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.717 INFO:teuthology.orchestra.run.smithi177.stdout:16 Sep 02:10:37 ntpd[18486]: ntpd: time slew -0.001864 s 2024-09-16T02:10:37.717 INFO:teuthology.orchestra.run.smithi177.stdout:ntpd: time slew -0.001864s 2024-09-16T02:10:37.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T02:10:37.717 INFO:teuthology.orchestra.run.smithi177.stderr:16 Sep 02:10:37 ntpd[18486]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T02:10:37.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.741 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:37.742 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:37.743 INFO:teuthology.orchestra.run.smithi195.stdout:16 Sep 02:10:37 ntpd[18465]: ntpd: time slew -0.000614 s 2024-09-16T02:10:37.743 INFO:teuthology.orchestra.run.smithi195.stdout:ntpd: time slew -0.000614s 2024-09-16T02:10:37.743 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T02:10:37.744 INFO:teuthology.orchestra.run.smithi195.stderr:16 Sep 02:10:37 ntpd[18465]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T02:10:37.763 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:10:37.763 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2024-09-16T02:10:37.763 INFO:teuthology.orchestra.run.smithi003.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.763 INFO:teuthology.orchestra.run.smithi003.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi003.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.764 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.951 INFO:teuthology.orchestra.run.smithi195.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:10:37.951 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================== 2024-09-16T02:10:37.951 INFO:teuthology.orchestra.run.smithi195.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.951 INFO:teuthology.orchestra.run.smithi195.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.952 INFO:teuthology.orchestra.run.smithi195.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:37.952 INFO:teuthology.orchestra.run.smithi195.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:38.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:38 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:39.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:39.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:39.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:39.714 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stdout:16 Sep 02:10:39 ntpd[18484]: ntpd: time slew -0.000667 s 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stdout:ntpd: time slew -0.000667s 2024-09-16T02:10:39.715 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T02:10:39.716 INFO:teuthology.orchestra.run.smithi118.stderr:16 Sep 02:10:39 ntpd[18484]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:39.772 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:10:39.773 INFO:teuthology.run_tasks:Running task install... 2024-09-16T02:10:39.781 DEBUG:teuthology.task.install:project ceph 2024-09-16T02:10:39.781 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-16T02:10:39.781 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-16T02:10:39.781 INFO:teuthology.task.install:Using flavor: default 2024-09-16T02:10:39.787 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T02:10:39.787 INFO:teuthology.task.install:extra packages: [] 2024-09-16T02:10:39.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-key list | grep Ceph 2024-09-16T02:10:39.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-key list | grep Ceph 2024-09-16T02:10:39.788 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-key list | grep Ceph 2024-09-16T02:10:39.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-key list | grep Ceph 2024-09-16T02:10:39.848 INFO:teuthology.orchestra.run.smithi177.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T02:10:39.849 INFO:teuthology.orchestra.run.smithi195.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T02:10:39.855 INFO:teuthology.orchestra.run.smithi003.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T02:10:39.870 INFO:teuthology.orchestra.run.smithi118.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T02:10:39.874 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T02:10:39.874 INFO:teuthology.orchestra.run.smithi177.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T02:10:39.875 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T02:10:39.876 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:10:39.888 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T02:10:39.888 INFO:teuthology.orchestra.run.smithi003.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T02:10:39.889 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T02:10:39.889 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:10:39.900 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T02:10:39.900 INFO:teuthology.orchestra.run.smithi118.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T02:10:39.900 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T02:10:39.900 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:10:39.912 INFO:teuthology.orchestra.run.smithi195.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T02:10:39.912 INFO:teuthology.orchestra.run.smithi195.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T02:10:39.913 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T02:10:39.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:10:40.032 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T02:10:40.032 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:40.046 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T02:10:40.046 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:40.057 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T02:10:40.057 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:40.067 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-16T02:10:40.067 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:40.145 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:10:40.145 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T02:10:40.158 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:10:40.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T02:10:40.160 DEBUG:teuthology.orchestra.run.smithi177:> sudo apt-get update 2024-09-16T02:10:40.165 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:10:40.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T02:10:40.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo apt-get update 2024-09-16T02:10:40.176 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:10:40.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T02:10:40.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo apt-get update 2024-09-16T02:10:40.191 DEBUG:teuthology.orchestra.run.smithi195:> sudo apt-get update 2024-09-16T02:10:40.329 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:10:40.329 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:10:40.340 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:10:40.342 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:10:40.351 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:10:40.351 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:10:40.353 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:10:40.356 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:10:40.357 INFO:teuthology.orchestra.run.smithi177.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:10:40.361 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:10:40.370 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:10:40.379 INFO:teuthology.orchestra.run.smithi003.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:10:40.383 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T02:10:40.398 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T02:10:40.405 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T02:10:40.406 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T02:10:40.410 INFO:teuthology.orchestra.run.smithi195.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T02:10:40.420 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T02:10:40.426 INFO:teuthology.orchestra.run.smithi177.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T02:10:40.427 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T02:10:40.431 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-16T02:10:40.441 INFO:teuthology.orchestra.run.smithi118.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T02:10:40.448 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T02:10:40.449 INFO:teuthology.orchestra.run.smithi003.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T02:10:40.453 INFO:teuthology.orchestra.run.smithi195.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T02:10:40.462 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T02:10:40.467 INFO:teuthology.orchestra.run.smithi118.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T02:10:40.470 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T02:10:40.474 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T02:10:40.474 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T02:10:40.481 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T02:10:40.489 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-16T02:10:40.501 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T02:10:40.503 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-16T02:10:40.559 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T02:10:40.641 INFO:teuthology.orchestra.run.smithi195.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T02:10:40.686 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 29.2 kB in 0s (84.9 kB/s) 2024-09-16T02:10:40.688 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 29.2 kB in 0s (88.0 kB/s) 2024-09-16T02:10:40.721 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 29.2 kB in 0s (82.3 kB/s) 2024-09-16T02:10:40.783 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 29.2 kB in 0s (71.2 kB/s) 2024-09-16T02:10:41.721 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T02:10:41.722 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T02:10:41.727 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T02:10:41.735 DEBUG:teuthology.orchestra.run.smithi177:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:41.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:41.743 DEBUG:teuthology.orchestra.run.smithi118:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:41.795 INFO:teuthology.orchestra.run.smithi003.stdout:Reading package lists... 2024-09-16T02:10:41.796 INFO:teuthology.orchestra.run.smithi118.stdout:Reading package lists... 2024-09-16T02:10:41.799 INFO:teuthology.orchestra.run.smithi177.stdout:Reading package lists... 2024-09-16T02:10:41.815 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T02:10:41.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:10:41.899 INFO:teuthology.orchestra.run.smithi195.stdout:Reading package lists... 2024-09-16T02:10:41.980 INFO:teuthology.orchestra.run.smithi118.stdout:Building dependency tree... 2024-09-16T02:10:41.981 INFO:teuthology.orchestra.run.smithi118.stdout:Reading state information... 2024-09-16T02:10:41.985 INFO:teuthology.orchestra.run.smithi003.stdout:Building dependency tree... 2024-09-16T02:10:41.986 INFO:teuthology.orchestra.run.smithi003.stdout:Reading state information... 2024-09-16T02:10:42.008 INFO:teuthology.orchestra.run.smithi177.stdout:Building dependency tree... 2024-09-16T02:10:42.009 INFO:teuthology.orchestra.run.smithi177.stdout:Reading state information... 2024-09-16T02:10:42.107 INFO:teuthology.orchestra.run.smithi195.stdout:Building dependency tree... 2024-09-16T02:10:42.107 INFO:teuthology.orchestra.run.smithi195.stdout:Reading state information... 2024-09-16T02:10:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:10:42.165 INFO:teuthology.orchestra.run.smithi118.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:10:42.165 INFO:teuthology.orchestra.run.smithi118.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:10:42.166 INFO:teuthology.orchestra.run.smithi118.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:10:42.166 INFO:teuthology.orchestra.run.smithi118.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:10:42.166 INFO:teuthology.orchestra.run.smithi118.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:10:42.166 INFO:teuthology.orchestra.run.smithi118.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:10:42.167 INFO:teuthology.orchestra.run.smithi118.stdout:The following additional packages will be installed: 2024-09-16T02:10:42.167 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T02:10:42.167 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T02:10:42.167 INFO:teuthology.orchestra.run.smithi118.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T02:10:42.168 INFO:teuthology.orchestra.run.smithi118.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T02:10:42.168 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T02:10:42.168 INFO:teuthology.orchestra.run.smithi118.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T02:10:42.168 INFO:teuthology.orchestra.run.smithi118.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T02:10:42.169 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet zip 2024-09-16T02:10:42.170 INFO:teuthology.orchestra.run.smithi118.stdout:Suggested packages: 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout:Recommended packages: 2024-09-16T02:10:42.171 INFO:teuthology.orchestra.run.smithi118.stdout: btrfs-tools 2024-09-16T02:10:42.180 INFO:teuthology.orchestra.run.smithi003.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:10:42.180 INFO:teuthology.orchestra.run.smithi003.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:10:42.180 INFO:teuthology.orchestra.run.smithi003.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:10:42.180 INFO:teuthology.orchestra.run.smithi003.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:10:42.181 INFO:teuthology.orchestra.run.smithi003.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:10:42.181 INFO:teuthology.orchestra.run.smithi003.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:10:42.181 INFO:teuthology.orchestra.run.smithi003.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:10:42.182 INFO:teuthology.orchestra.run.smithi003.stdout:The following additional packages will be installed: 2024-09-16T02:10:42.182 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T02:10:42.182 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T02:10:42.182 INFO:teuthology.orchestra.run.smithi003.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T02:10:42.182 INFO:teuthology.orchestra.run.smithi003.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T02:10:42.183 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T02:10:42.183 INFO:teuthology.orchestra.run.smithi003.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T02:10:42.183 INFO:teuthology.orchestra.run.smithi003.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T02:10:42.183 INFO:teuthology.orchestra.run.smithi003.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T02:10:42.184 INFO:teuthology.orchestra.run.smithi003.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet zip 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout:Suggested packages: 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T02:10:42.185 INFO:teuthology.orchestra.run.smithi003.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T02:10:42.186 INFO:teuthology.orchestra.run.smithi003.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T02:10:42.186 INFO:teuthology.orchestra.run.smithi003.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T02:10:42.186 INFO:teuthology.orchestra.run.smithi003.stdout:Recommended packages: 2024-09-16T02:10:42.186 INFO:teuthology.orchestra.run.smithi003.stdout: btrfs-tools 2024-09-16T02:10:42.213 INFO:teuthology.orchestra.run.smithi177.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:10:42.213 INFO:teuthology.orchestra.run.smithi177.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:10:42.213 INFO:teuthology.orchestra.run.smithi177.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:10:42.213 INFO:teuthology.orchestra.run.smithi177.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout:The following additional packages will be installed: 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T02:10:42.215 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T02:10:42.216 INFO:teuthology.orchestra.run.smithi177.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T02:10:42.217 INFO:teuthology.orchestra.run.smithi177.stdout: xmlstarlet zip 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout:Suggested packages: 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout:Recommended packages: 2024-09-16T02:10:42.218 INFO:teuthology.orchestra.run.smithi177.stdout: btrfs-tools 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout:The following NEW packages will be installed: 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T02:10:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T02:10:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T02:10:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T02:10:42.289 INFO:teuthology.orchestra.run.smithi003.stdout:The following NEW packages will be installed: 2024-09-16T02:10:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T02:10:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T02:10:42.290 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T02:10:42.291 INFO:teuthology.orchestra.run.smithi003.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T02:10:42.292 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T02:10:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T02:10:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T02:10:42.293 INFO:teuthology.orchestra.run.smithi003.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T02:10:42.301 INFO:teuthology.orchestra.run.smithi195.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T02:10:42.301 INFO:teuthology.orchestra.run.smithi195.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T02:10:42.301 INFO:teuthology.orchestra.run.smithi195.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T02:10:42.302 INFO:teuthology.orchestra.run.smithi195.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T02:10:42.302 INFO:teuthology.orchestra.run.smithi195.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T02:10:42.302 INFO:teuthology.orchestra.run.smithi195.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T02:10:42.302 INFO:teuthology.orchestra.run.smithi195.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout:The following additional packages will be installed: 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T02:10:42.303 INFO:teuthology.orchestra.run.smithi195.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-16T02:10:42.304 INFO:teuthology.orchestra.run.smithi195.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T02:10:42.305 INFO:teuthology.orchestra.run.smithi195.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout: xmlstarlet zip 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout:Suggested packages: 2024-09-16T02:10:42.306 INFO:teuthology.orchestra.run.smithi195.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout:Recommended packages: 2024-09-16T02:10:42.307 INFO:teuthology.orchestra.run.smithi195.stdout: btrfs-tools 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout:The following NEW packages will be installed: 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T02:10:42.317 INFO:teuthology.orchestra.run.smithi177.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T02:10:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T02:10:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T02:10:42.318 INFO:teuthology.orchestra.run.smithi177.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T02:10:42.319 INFO:teuthology.orchestra.run.smithi177.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T02:10:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T02:10:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T02:10:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T02:10:42.320 INFO:teuthology.orchestra.run.smithi177.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T02:10:42.331 INFO:teuthology.orchestra.run.smithi118.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T02:10:42.331 INFO:teuthology.orchestra.run.smithi118.stdout:Need to get 244 MB of archives. 2024-09-16T02:10:42.331 INFO:teuthology.orchestra.run.smithi118.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T02:10:42.331 INFO:teuthology.orchestra.run.smithi118.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout:The following NEW packages will be installed: 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-16T02:10:42.411 INFO:teuthology.orchestra.run.smithi195.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-16T02:10:42.412 INFO:teuthology.orchestra.run.smithi195.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-16T02:10:42.412 INFO:teuthology.orchestra.run.smithi195.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-16T02:10:42.412 INFO:teuthology.orchestra.run.smithi195.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-16T02:10:42.413 INFO:teuthology.orchestra.run.smithi195.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-16T02:10:42.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-16T02:10:42.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-16T02:10:42.414 INFO:teuthology.orchestra.run.smithi195.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-16T02:10:42.414 INFO:teuthology.orchestra.run.smithi195.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-16T02:10:42.414 INFO:teuthology.orchestra.run.smithi118.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T02:10:42.419 INFO:teuthology.orchestra.run.smithi118.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T02:10:42.421 INFO:teuthology.orchestra.run.smithi118.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T02:10:42.437 INFO:teuthology.orchestra.run.smithi003.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T02:10:42.437 INFO:teuthology.orchestra.run.smithi003.stdout:Need to get 244 MB of archives. 2024-09-16T02:10:42.437 INFO:teuthology.orchestra.run.smithi003.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T02:10:42.437 INFO:teuthology.orchestra.run.smithi003.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T02:10:42.438 INFO:teuthology.orchestra.run.smithi118.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T02:10:42.461 INFO:teuthology.orchestra.run.smithi177.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T02:10:42.462 INFO:teuthology.orchestra.run.smithi177.stdout:Need to get 244 MB of archives. 2024-09-16T02:10:42.462 INFO:teuthology.orchestra.run.smithi177.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T02:10:42.462 INFO:teuthology.orchestra.run.smithi177.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T02:10:42.481 INFO:teuthology.orchestra.run.smithi003.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T02:10:42.496 INFO:teuthology.orchestra.run.smithi118.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T02:10:42.501 INFO:teuthology.orchestra.run.smithi118.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T02:10:42.506 INFO:teuthology.orchestra.run.smithi177.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T02:10:42.513 INFO:teuthology.orchestra.run.smithi118.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T02:10:42.517 INFO:teuthology.orchestra.run.smithi118.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T02:10:42.518 INFO:teuthology.orchestra.run.smithi118.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T02:10:42.519 INFO:teuthology.orchestra.run.smithi118.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T02:10:42.520 INFO:teuthology.orchestra.run.smithi118.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T02:10:42.528 INFO:teuthology.orchestra.run.smithi118.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T02:10:42.530 INFO:teuthology.orchestra.run.smithi118.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T02:10:42.532 INFO:teuthology.orchestra.run.smithi118.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T02:10:42.535 INFO:teuthology.orchestra.run.smithi118.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T02:10:42.536 INFO:teuthology.orchestra.run.smithi118.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T02:10:42.553 INFO:teuthology.orchestra.run.smithi118.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T02:10:42.555 INFO:teuthology.orchestra.run.smithi118.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T02:10:42.562 INFO:teuthology.orchestra.run.smithi118.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T02:10:42.563 INFO:teuthology.orchestra.run.smithi118.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T02:10:42.563 INFO:teuthology.orchestra.run.smithi195.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T02:10:42.563 INFO:teuthology.orchestra.run.smithi195.stdout:Need to get 244 MB of archives. 2024-09-16T02:10:42.564 INFO:teuthology.orchestra.run.smithi195.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-16T02:10:42.564 INFO:teuthology.orchestra.run.smithi195.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-16T02:10:42.564 INFO:teuthology.orchestra.run.smithi118.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T02:10:42.565 INFO:teuthology.orchestra.run.smithi118.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T02:10:42.565 INFO:teuthology.orchestra.run.smithi118.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T02:10:42.565 INFO:teuthology.orchestra.run.smithi118.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T02:10:42.569 INFO:teuthology.orchestra.run.smithi118.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T02:10:42.570 INFO:teuthology.orchestra.run.smithi118.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T02:10:42.586 INFO:teuthology.orchestra.run.smithi118.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T02:10:42.587 INFO:teuthology.orchestra.run.smithi118.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T02:10:42.589 INFO:teuthology.orchestra.run.smithi118.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T02:10:42.590 INFO:teuthology.orchestra.run.smithi118.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T02:10:42.590 INFO:teuthology.orchestra.run.smithi118.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T02:10:42.590 INFO:teuthology.orchestra.run.smithi118.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T02:10:42.592 INFO:teuthology.orchestra.run.smithi118.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T02:10:42.592 INFO:teuthology.orchestra.run.smithi118.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T02:10:42.602 INFO:teuthology.orchestra.run.smithi195.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T02:10:42.603 INFO:teuthology.orchestra.run.smithi118.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T02:10:42.605 INFO:teuthology.orchestra.run.smithi118.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T02:10:42.621 INFO:teuthology.orchestra.run.smithi118.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T02:10:42.621 INFO:teuthology.orchestra.run.smithi118.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T02:10:42.627 INFO:teuthology.orchestra.run.smithi118.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T02:10:42.627 INFO:teuthology.orchestra.run.smithi118.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T02:10:42.627 INFO:teuthology.orchestra.run.smithi118.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T02:10:42.628 INFO:teuthology.orchestra.run.smithi118.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T02:10:42.629 INFO:teuthology.orchestra.run.smithi118.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T02:10:42.631 INFO:teuthology.orchestra.run.smithi118.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T02:10:42.632 INFO:teuthology.orchestra.run.smithi118.stdout:Get:46 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T02:10:42.638 INFO:teuthology.orchestra.run.smithi118.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T02:10:42.639 INFO:teuthology.orchestra.run.smithi118.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T02:10:42.649 INFO:teuthology.orchestra.run.smithi118.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T02:10:42.651 INFO:teuthology.orchestra.run.smithi118.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T02:10:42.652 INFO:teuthology.orchestra.run.smithi118.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T02:10:42.653 INFO:teuthology.orchestra.run.smithi118.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T02:10:42.655 INFO:teuthology.orchestra.run.smithi118.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T02:10:42.682 INFO:teuthology.orchestra.run.smithi003.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T02:10:42.705 INFO:teuthology.orchestra.run.smithi118.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T02:10:42.706 INFO:teuthology.orchestra.run.smithi118.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T02:10:42.707 INFO:teuthology.orchestra.run.smithi118.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T02:10:42.708 INFO:teuthology.orchestra.run.smithi118.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T02:10:42.709 INFO:teuthology.orchestra.run.smithi118.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T02:10:42.725 INFO:teuthology.orchestra.run.smithi118.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T02:10:42.727 INFO:teuthology.orchestra.run.smithi118.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T02:10:42.727 INFO:teuthology.orchestra.run.smithi118.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T02:10:42.731 INFO:teuthology.orchestra.run.smithi118.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T02:10:42.744 INFO:teuthology.orchestra.run.smithi118.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T02:10:42.744 INFO:teuthology.orchestra.run.smithi118.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T02:10:42.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T02:10:42.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T02:10:42.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T02:10:42.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T02:10:42.745 INFO:teuthology.orchestra.run.smithi118.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T02:10:42.752 INFO:teuthology.orchestra.run.smithi118.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T02:10:42.752 INFO:teuthology.orchestra.run.smithi118.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T02:10:42.757 INFO:teuthology.orchestra.run.smithi118.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T02:10:42.758 INFO:teuthology.orchestra.run.smithi118.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T02:10:42.759 INFO:teuthology.orchestra.run.smithi118.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T02:10:42.760 INFO:teuthology.orchestra.run.smithi118.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T02:10:42.763 INFO:teuthology.orchestra.run.smithi118.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T02:10:42.776 INFO:teuthology.orchestra.run.smithi118.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T02:10:42.777 INFO:teuthology.orchestra.run.smithi118.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T02:10:42.778 INFO:teuthology.orchestra.run.smithi118.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T02:10:42.781 INFO:teuthology.orchestra.run.smithi118.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T02:10:42.783 INFO:teuthology.orchestra.run.smithi118.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T02:10:42.784 INFO:teuthology.orchestra.run.smithi118.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T02:10:42.784 INFO:teuthology.orchestra.run.smithi118.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T02:10:42.791 INFO:teuthology.orchestra.run.smithi003.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T02:10:42.793 INFO:teuthology.orchestra.run.smithi118.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T02:10:42.794 INFO:teuthology.orchestra.run.smithi118.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T02:10:42.795 INFO:teuthology.orchestra.run.smithi118.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T02:10:42.797 INFO:teuthology.orchestra.run.smithi118.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T02:10:42.810 INFO:teuthology.orchestra.run.smithi118.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T02:10:42.811 INFO:teuthology.orchestra.run.smithi118.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T02:10:42.811 INFO:teuthology.orchestra.run.smithi118.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T02:10:42.812 INFO:teuthology.orchestra.run.smithi118.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T02:10:42.813 INFO:teuthology.orchestra.run.smithi118.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T02:10:42.819 INFO:teuthology.orchestra.run.smithi118.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T02:10:42.819 INFO:teuthology.orchestra.run.smithi118.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T02:10:42.827 INFO:teuthology.orchestra.run.smithi118.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T02:10:42.829 INFO:teuthology.orchestra.run.smithi118.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T02:10:42.829 INFO:teuthology.orchestra.run.smithi118.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T02:10:42.903 INFO:teuthology.orchestra.run.smithi003.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T02:10:42.916 INFO:teuthology.orchestra.run.smithi003.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T02:10:42.925 INFO:teuthology.orchestra.run.smithi003.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T02:10:42.929 INFO:teuthology.orchestra.run.smithi177.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T02:10:42.932 INFO:teuthology.orchestra.run.smithi003.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T02:10:42.933 INFO:teuthology.orchestra.run.smithi003.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T02:10:42.937 INFO:teuthology.orchestra.run.smithi003.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T02:10:42.939 INFO:teuthology.orchestra.run.smithi003.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T02:10:42.942 INFO:teuthology.orchestra.run.smithi177.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T02:10:42.946 INFO:teuthology.orchestra.run.smithi003.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T02:10:42.996 INFO:teuthology.orchestra.run.smithi003.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T02:10:43.023 INFO:teuthology.orchestra.run.smithi177.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T02:10:43.025 INFO:teuthology.orchestra.run.smithi195.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T02:10:43.038 INFO:teuthology.orchestra.run.smithi195.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T02:10:43.055 INFO:teuthology.orchestra.run.smithi177.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T02:10:43.086 INFO:teuthology.orchestra.run.smithi118.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T02:10:43.088 INFO:teuthology.orchestra.run.smithi118.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T02:10:43.096 INFO:teuthology.orchestra.run.smithi118.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T02:10:43.119 INFO:teuthology.orchestra.run.smithi195.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T02:10:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T02:10:43.200 INFO:teuthology.orchestra.run.smithi003.stdout:Get:15 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T02:10:43.209 INFO:teuthology.orchestra.run.smithi003.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T02:10:43.225 INFO:teuthology.orchestra.run.smithi003.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T02:10:43.237 INFO:teuthology.orchestra.run.smithi003.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T02:10:43.251 INFO:teuthology.orchestra.run.smithi177.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T02:10:43.262 INFO:teuthology.orchestra.run.smithi177.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T02:10:43.266 INFO:teuthology.orchestra.run.smithi003.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T02:10:43.275 INFO:teuthology.orchestra.run.smithi003.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T02:10:43.277 INFO:teuthology.orchestra.run.smithi003.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T02:10:43.278 INFO:teuthology.orchestra.run.smithi003.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T02:10:43.279 INFO:teuthology.orchestra.run.smithi003.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T02:10:43.292 INFO:teuthology.orchestra.run.smithi177.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T02:10:43.297 INFO:teuthology.orchestra.run.smithi003.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T02:10:43.301 INFO:teuthology.orchestra.run.smithi177.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T02:10:43.301 INFO:teuthology.orchestra.run.smithi003.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T02:10:43.303 INFO:teuthology.orchestra.run.smithi177.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T02:10:43.304 INFO:teuthology.orchestra.run.smithi177.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T02:10:43.305 INFO:teuthology.orchestra.run.smithi177.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T02:10:43.308 INFO:teuthology.orchestra.run.smithi003.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T02:10:43.323 INFO:teuthology.orchestra.run.smithi177.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T02:10:43.327 INFO:teuthology.orchestra.run.smithi177.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T02:10:43.334 INFO:teuthology.orchestra.run.smithi177.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T02:10:43.348 INFO:teuthology.orchestra.run.smithi195.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T02:10:43.359 INFO:teuthology.orchestra.run.smithi195.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T02:10:43.388 INFO:teuthology.orchestra.run.smithi003.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T02:10:43.388 INFO:teuthology.orchestra.run.smithi003.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T02:10:43.389 INFO:teuthology.orchestra.run.smithi195.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T02:10:43.393 INFO:teuthology.orchestra.run.smithi003.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T02:10:43.397 INFO:teuthology.orchestra.run.smithi003.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T02:10:43.398 INFO:teuthology.orchestra.run.smithi195.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-16T02:10:43.400 INFO:teuthology.orchestra.run.smithi195.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T02:10:43.401 INFO:teuthology.orchestra.run.smithi195.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T02:10:43.401 INFO:teuthology.orchestra.run.smithi003.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T02:10:43.402 INFO:teuthology.orchestra.run.smithi003.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T02:10:43.402 INFO:teuthology.orchestra.run.smithi195.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T02:10:43.403 INFO:teuthology.orchestra.run.smithi003.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T02:10:43.405 INFO:teuthology.orchestra.run.smithi003.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T02:10:43.405 INFO:teuthology.orchestra.run.smithi003.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T02:10:43.413 INFO:teuthology.orchestra.run.smithi177.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T02:10:43.414 INFO:teuthology.orchestra.run.smithi177.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T02:10:43.418 INFO:teuthology.orchestra.run.smithi177.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T02:10:43.420 INFO:teuthology.orchestra.run.smithi195.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-16T02:10:43.423 INFO:teuthology.orchestra.run.smithi177.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T02:10:43.425 INFO:teuthology.orchestra.run.smithi195.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-16T02:10:43.426 INFO:teuthology.orchestra.run.smithi177.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T02:10:43.427 INFO:teuthology.orchestra.run.smithi177.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T02:10:43.428 INFO:teuthology.orchestra.run.smithi177.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T02:10:43.430 INFO:teuthology.orchestra.run.smithi177.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T02:10:43.430 INFO:teuthology.orchestra.run.smithi195.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-16T02:10:43.431 INFO:teuthology.orchestra.run.smithi177.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T02:10:43.471 INFO:teuthology.orchestra.run.smithi003.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T02:10:43.488 INFO:teuthology.orchestra.run.smithi003.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T02:10:43.489 INFO:teuthology.orchestra.run.smithi003.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T02:10:43.490 INFO:teuthology.orchestra.run.smithi003.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T02:10:43.491 INFO:teuthology.orchestra.run.smithi003.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T02:10:43.496 INFO:teuthology.orchestra.run.smithi177.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T02:10:43.496 INFO:teuthology.orchestra.run.smithi003.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T02:10:43.497 INFO:teuthology.orchestra.run.smithi003.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T02:10:43.511 INFO:teuthology.orchestra.run.smithi195.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-16T02:10:43.511 INFO:teuthology.orchestra.run.smithi195.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-16T02:10:43.513 INFO:teuthology.orchestra.run.smithi177.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T02:10:43.514 INFO:teuthology.orchestra.run.smithi177.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T02:10:43.515 INFO:teuthology.orchestra.run.smithi177.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T02:10:43.516 INFO:teuthology.orchestra.run.smithi177.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T02:10:43.516 INFO:teuthology.orchestra.run.smithi195.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-16T02:10:43.520 INFO:teuthology.orchestra.run.smithi195.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-16T02:10:43.521 INFO:teuthology.orchestra.run.smithi177.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T02:10:43.523 INFO:teuthology.orchestra.run.smithi195.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T02:10:43.525 INFO:teuthology.orchestra.run.smithi195.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T02:10:43.525 INFO:teuthology.orchestra.run.smithi195.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T02:10:43.527 INFO:teuthology.orchestra.run.smithi195.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T02:10:43.528 INFO:teuthology.orchestra.run.smithi195.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T02:10:43.553 INFO:teuthology.orchestra.run.smithi003.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T02:10:43.553 INFO:teuthology.orchestra.run.smithi003.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T02:10:43.556 INFO:teuthology.orchestra.run.smithi003.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T02:10:43.564 INFO:teuthology.orchestra.run.smithi177.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T02:10:43.577 INFO:teuthology.orchestra.run.smithi177.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T02:10:43.578 INFO:teuthology.orchestra.run.smithi177.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T02:10:43.579 INFO:teuthology.orchestra.run.smithi177.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T02:10:43.581 INFO:teuthology.orchestra.run.smithi177.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T02:10:43.594 INFO:teuthology.orchestra.run.smithi195.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T02:10:43.613 INFO:teuthology.orchestra.run.smithi195.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T02:10:43.613 INFO:teuthology.orchestra.run.smithi195.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T02:10:43.614 INFO:teuthology.orchestra.run.smithi195.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T02:10:43.615 INFO:teuthology.orchestra.run.smithi195.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T02:10:43.621 INFO:teuthology.orchestra.run.smithi195.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T02:10:43.622 INFO:teuthology.orchestra.run.smithi195.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T02:10:43.634 INFO:teuthology.orchestra.run.smithi003.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T02:10:43.634 INFO:teuthology.orchestra.run.smithi003.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T02:10:43.637 INFO:teuthology.orchestra.run.smithi003.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T02:10:43.639 INFO:teuthology.orchestra.run.smithi003.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T02:10:43.639 INFO:teuthology.orchestra.run.smithi003.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T02:10:43.651 INFO:teuthology.orchestra.run.smithi003.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T02:10:43.659 INFO:teuthology.orchestra.run.smithi177.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T02:10:43.659 INFO:teuthology.orchestra.run.smithi177.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T02:10:43.661 INFO:teuthology.orchestra.run.smithi177.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T02:10:43.663 INFO:teuthology.orchestra.run.smithi177.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T02:10:43.664 INFO:teuthology.orchestra.run.smithi177.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T02:10:43.676 INFO:teuthology.orchestra.run.smithi195.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T02:10:43.677 INFO:teuthology.orchestra.run.smithi195.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T02:10:43.679 INFO:teuthology.orchestra.run.smithi195.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T02:10:43.693 INFO:teuthology.orchestra.run.smithi177.stdout:Get:42 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T02:10:43.715 INFO:teuthology.orchestra.run.smithi003.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T02:10:43.716 INFO:teuthology.orchestra.run.smithi003.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T02:10:43.717 INFO:teuthology.orchestra.run.smithi003.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T02:10:43.720 INFO:teuthology.orchestra.run.smithi003.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T02:10:43.740 INFO:teuthology.orchestra.run.smithi177.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T02:10:43.741 INFO:teuthology.orchestra.run.smithi177.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T02:10:43.742 INFO:teuthology.orchestra.run.smithi177.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T02:10:43.743 INFO:teuthology.orchestra.run.smithi177.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T02:10:43.745 INFO:teuthology.orchestra.run.smithi177.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T02:10:43.757 INFO:teuthology.orchestra.run.smithi195.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T02:10:43.758 INFO:teuthology.orchestra.run.smithi177.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T02:10:43.759 INFO:teuthology.orchestra.run.smithi195.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T02:10:43.759 INFO:teuthology.orchestra.run.smithi177.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T02:10:43.761 INFO:teuthology.orchestra.run.smithi195.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T02:10:43.762 INFO:teuthology.orchestra.run.smithi195.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T02:10:43.763 INFO:teuthology.orchestra.run.smithi177.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T02:10:43.764 INFO:teuthology.orchestra.run.smithi195.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T02:10:43.774 INFO:teuthology.orchestra.run.smithi195.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T02:10:43.797 INFO:teuthology.orchestra.run.smithi003.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T02:10:43.798 INFO:teuthology.orchestra.run.smithi003.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T02:10:43.801 INFO:teuthology.orchestra.run.smithi003.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T02:10:43.803 INFO:teuthology.orchestra.run.smithi003.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T02:10:43.807 INFO:teuthology.orchestra.run.smithi003.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T02:10:43.808 INFO:teuthology.orchestra.run.smithi003.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T02:10:43.822 INFO:teuthology.orchestra.run.smithi177.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T02:10:43.823 INFO:teuthology.orchestra.run.smithi177.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T02:10:43.823 INFO:teuthology.orchestra.run.smithi177.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T02:10:43.826 INFO:teuthology.orchestra.run.smithi177.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T02:10:43.827 INFO:teuthology.orchestra.run.smithi177.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T02:10:43.829 INFO:teuthology.orchestra.run.smithi177.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T02:10:43.832 INFO:teuthology.orchestra.run.smithi177.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T02:10:43.839 INFO:teuthology.orchestra.run.smithi195.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T02:10:43.840 INFO:teuthology.orchestra.run.smithi195.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T02:10:43.841 INFO:teuthology.orchestra.run.smithi195.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T02:10:43.843 INFO:teuthology.orchestra.run.smithi195.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T02:10:43.880 INFO:teuthology.orchestra.run.smithi003.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T02:10:43.884 INFO:teuthology.orchestra.run.smithi003.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T02:10:43.903 INFO:teuthology.orchestra.run.smithi177.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T02:10:43.904 INFO:teuthology.orchestra.run.smithi177.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T02:10:43.909 INFO:teuthology.orchestra.run.smithi177.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T02:10:43.920 INFO:teuthology.orchestra.run.smithi195.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T02:10:43.922 INFO:teuthology.orchestra.run.smithi195.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T02:10:43.925 INFO:teuthology.orchestra.run.smithi195.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T02:10:43.926 INFO:teuthology.orchestra.run.smithi195.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T02:10:43.931 INFO:teuthology.orchestra.run.smithi195.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T02:10:43.932 INFO:teuthology.orchestra.run.smithi195.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T02:10:44.002 INFO:teuthology.orchestra.run.smithi195.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T02:10:44.007 INFO:teuthology.orchestra.run.smithi195.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T02:10:44.049 INFO:teuthology.orchestra.run.smithi003.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T02:10:44.050 INFO:teuthology.orchestra.run.smithi003.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T02:10:44.053 INFO:teuthology.orchestra.run.smithi003.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T02:10:44.053 INFO:teuthology.orchestra.run.smithi003.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T02:10:44.055 INFO:teuthology.orchestra.run.smithi003.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T02:10:44.072 INFO:teuthology.orchestra.run.smithi177.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T02:10:44.074 INFO:teuthology.orchestra.run.smithi177.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T02:10:44.076 INFO:teuthology.orchestra.run.smithi177.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T02:10:44.077 INFO:teuthology.orchestra.run.smithi177.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T02:10:44.079 INFO:teuthology.orchestra.run.smithi177.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T02:10:44.088 INFO:teuthology.orchestra.run.smithi003.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T02:10:44.091 INFO:teuthology.orchestra.run.smithi003.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T02:10:44.091 INFO:teuthology.orchestra.run.smithi003.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T02:10:44.116 INFO:teuthology.orchestra.run.smithi177.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T02:10:44.118 INFO:teuthology.orchestra.run.smithi177.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T02:10:44.119 INFO:teuthology.orchestra.run.smithi177.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T02:10:44.136 INFO:teuthology.orchestra.run.smithi195.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-16T02:10:44.150 INFO:teuthology.orchestra.run.smithi003.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T02:10:44.150 INFO:teuthology.orchestra.run.smithi003.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T02:10:44.175 INFO:teuthology.orchestra.run.smithi195.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T02:10:44.176 INFO:teuthology.orchestra.run.smithi177.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T02:10:44.176 INFO:teuthology.orchestra.run.smithi177.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T02:10:44.177 INFO:teuthology.orchestra.run.smithi195.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T02:10:44.180 INFO:teuthology.orchestra.run.smithi195.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T02:10:44.181 INFO:teuthology.orchestra.run.smithi195.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T02:10:44.183 INFO:teuthology.orchestra.run.smithi195.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T02:10:44.215 INFO:teuthology.orchestra.run.smithi003.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T02:10:44.216 INFO:teuthology.orchestra.run.smithi003.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T02:10:44.217 INFO:teuthology.orchestra.run.smithi003.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T02:10:44.218 INFO:teuthology.orchestra.run.smithi003.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T02:10:44.218 INFO:teuthology.orchestra.run.smithi003.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T02:10:44.219 INFO:teuthology.orchestra.run.smithi003.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T02:10:44.220 INFO:teuthology.orchestra.run.smithi195.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T02:10:44.223 INFO:teuthology.orchestra.run.smithi195.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T02:10:44.223 INFO:teuthology.orchestra.run.smithi195.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T02:10:44.225 INFO:teuthology.orchestra.run.smithi003.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T02:10:44.236 INFO:teuthology.orchestra.run.smithi118.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T02:10:44.238 INFO:teuthology.orchestra.run.smithi177.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T02:10:44.239 INFO:teuthology.orchestra.run.smithi177.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T02:10:44.240 INFO:teuthology.orchestra.run.smithi177.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T02:10:44.241 INFO:teuthology.orchestra.run.smithi177.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T02:10:44.242 INFO:teuthology.orchestra.run.smithi177.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T02:10:44.243 INFO:teuthology.orchestra.run.smithi177.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T02:10:44.250 INFO:teuthology.orchestra.run.smithi177.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T02:10:44.280 INFO:teuthology.orchestra.run.smithi195.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T02:10:44.281 INFO:teuthology.orchestra.run.smithi195.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T02:10:44.296 INFO:teuthology.orchestra.run.smithi003.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T02:10:44.299 INFO:teuthology.orchestra.run.smithi003.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T02:10:44.301 INFO:teuthology.orchestra.run.smithi003.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T02:10:44.320 INFO:teuthology.orchestra.run.smithi177.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T02:10:44.323 INFO:teuthology.orchestra.run.smithi177.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T02:10:44.325 INFO:teuthology.orchestra.run.smithi177.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T02:10:44.341 INFO:teuthology.orchestra.run.smithi118.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T02:10:44.342 INFO:teuthology.orchestra.run.smithi195.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T02:10:44.343 INFO:teuthology.orchestra.run.smithi195.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T02:10:44.344 INFO:teuthology.orchestra.run.smithi195.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T02:10:44.345 INFO:teuthology.orchestra.run.smithi195.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T02:10:44.346 INFO:teuthology.orchestra.run.smithi118.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T02:10:44.346 INFO:teuthology.orchestra.run.smithi195.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T02:10:44.347 INFO:teuthology.orchestra.run.smithi195.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T02:10:44.349 INFO:teuthology.orchestra.run.smithi118.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T02:10:44.355 INFO:teuthology.orchestra.run.smithi195.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T02:10:44.378 INFO:teuthology.orchestra.run.smithi003.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T02:10:44.384 INFO:teuthology.orchestra.run.smithi003.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T02:10:44.386 INFO:teuthology.orchestra.run.smithi003.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T02:10:44.388 INFO:teuthology.orchestra.run.smithi003.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T02:10:44.401 INFO:teuthology.orchestra.run.smithi177.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T02:10:44.407 INFO:teuthology.orchestra.run.smithi177.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T02:10:44.409 INFO:teuthology.orchestra.run.smithi177.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T02:10:44.411 INFO:teuthology.orchestra.run.smithi177.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T02:10:44.411 INFO:teuthology.orchestra.run.smithi118.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T02:10:44.422 INFO:teuthology.orchestra.run.smithi177.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T02:10:44.423 INFO:teuthology.orchestra.run.smithi177.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T02:10:44.424 INFO:teuthology.orchestra.run.smithi195.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T02:10:44.428 INFO:teuthology.orchestra.run.smithi195.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T02:10:44.429 INFO:teuthology.orchestra.run.smithi195.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T02:10:44.431 INFO:teuthology.orchestra.run.smithi177.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T02:10:44.464 INFO:teuthology.orchestra.run.smithi003.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T02:10:44.465 INFO:teuthology.orchestra.run.smithi003.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T02:10:44.483 INFO:teuthology.orchestra.run.smithi177.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T02:10:44.484 INFO:teuthology.orchestra.run.smithi177.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T02:10:44.485 INFO:teuthology.orchestra.run.smithi177.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T02:10:44.505 INFO:teuthology.orchestra.run.smithi195.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T02:10:44.512 INFO:teuthology.orchestra.run.smithi195.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-16T02:10:44.515 INFO:teuthology.orchestra.run.smithi195.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-16T02:10:44.517 INFO:teuthology.orchestra.run.smithi195.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T02:10:44.527 INFO:teuthology.orchestra.run.smithi195.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-16T02:10:44.529 INFO:teuthology.orchestra.run.smithi195.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T02:10:44.536 INFO:teuthology.orchestra.run.smithi195.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T02:10:44.551 INFO:teuthology.orchestra.run.smithi003.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T02:10:44.551 INFO:teuthology.orchestra.run.smithi003.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T02:10:44.551 INFO:teuthology.orchestra.run.smithi003.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T02:10:44.553 INFO:teuthology.orchestra.run.smithi003.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T02:10:44.554 INFO:teuthology.orchestra.run.smithi003.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T02:10:44.558 INFO:teuthology.orchestra.run.smithi003.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T02:10:44.560 INFO:teuthology.orchestra.run.smithi003.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T02:10:44.564 INFO:teuthology.orchestra.run.smithi177.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T02:10:44.566 INFO:teuthology.orchestra.run.smithi177.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T02:10:44.577 INFO:teuthology.orchestra.run.smithi003.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T02:10:44.583 INFO:teuthology.orchestra.run.smithi177.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T02:10:44.584 INFO:teuthology.orchestra.run.smithi177.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T02:10:44.585 INFO:teuthology.orchestra.run.smithi177.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T02:10:44.587 INFO:teuthology.orchestra.run.smithi195.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T02:10:44.589 INFO:teuthology.orchestra.run.smithi195.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T02:10:44.589 INFO:teuthology.orchestra.run.smithi177.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T02:10:44.590 INFO:teuthology.orchestra.run.smithi195.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T02:10:44.591 INFO:teuthology.orchestra.run.smithi177.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T02:10:44.604 INFO:teuthology.orchestra.run.smithi118.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T02:10:44.622 INFO:teuthology.orchestra.run.smithi003.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T02:10:44.623 INFO:teuthology.orchestra.run.smithi003.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T02:10:44.656 INFO:teuthology.orchestra.run.smithi003.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T02:10:44.669 INFO:teuthology.orchestra.run.smithi003.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T02:10:44.669 INFO:teuthology.orchestra.run.smithi195.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T02:10:44.671 INFO:teuthology.orchestra.run.smithi195.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T02:10:44.689 INFO:teuthology.orchestra.run.smithi003.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T02:10:44.690 INFO:teuthology.orchestra.run.smithi195.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T02:10:44.691 INFO:teuthology.orchestra.run.smithi195.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T02:10:44.692 INFO:teuthology.orchestra.run.smithi195.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T02:10:44.697 INFO:teuthology.orchestra.run.smithi195.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T02:10:44.699 INFO:teuthology.orchestra.run.smithi195.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T02:10:44.703 INFO:teuthology.orchestra.run.smithi003.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T02:10:44.705 INFO:teuthology.orchestra.run.smithi003.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T02:10:44.708 INFO:teuthology.orchestra.run.smithi195.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-16T02:10:44.763 INFO:teuthology.orchestra.run.smithi003.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T02:10:44.854 INFO:teuthology.orchestra.run.smithi195.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-16T02:10:44.929 INFO:teuthology.orchestra.run.smithi195.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-16T02:10:44.930 INFO:teuthology.orchestra.run.smithi195.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-16T02:10:44.934 INFO:teuthology.orchestra.run.smithi195.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-16T02:10:44.936 INFO:teuthology.orchestra.run.smithi195.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-16T02:10:45.009 INFO:teuthology.orchestra.run.smithi195.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-16T02:10:45.040 INFO:teuthology.orchestra.run.smithi003.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T02:10:45.078 INFO:teuthology.orchestra.run.smithi177.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T02:10:45.081 INFO:teuthology.orchestra.run.smithi177.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T02:10:45.157 INFO:teuthology.orchestra.run.smithi177.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T02:10:45.519 INFO:teuthology.orchestra.run.smithi118.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T02:10:45.520 INFO:teuthology.orchestra.run.smithi118.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T02:10:45.539 INFO:teuthology.orchestra.run.smithi118.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T02:10:45.580 INFO:teuthology.orchestra.run.smithi118.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T02:10:45.590 INFO:teuthology.orchestra.run.smithi118.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T02:10:45.592 INFO:teuthology.orchestra.run.smithi118.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T02:10:46.210 INFO:teuthology.orchestra.run.smithi195.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-16T02:10:46.220 INFO:teuthology.orchestra.run.smithi195.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-16T02:10:46.294 INFO:teuthology.orchestra.run.smithi195.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-16T02:10:46.403 INFO:teuthology.orchestra.run.smithi003.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T02:10:46.404 INFO:teuthology.orchestra.run.smithi003.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T02:10:46.422 INFO:teuthology.orchestra.run.smithi003.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T02:10:46.512 INFO:teuthology.orchestra.run.smithi003.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T02:10:46.578 INFO:teuthology.orchestra.run.smithi003.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T02:10:46.581 INFO:teuthology.orchestra.run.smithi003.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T02:10:47.909 INFO:teuthology.orchestra.run.smithi118.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T02:10:48.195 INFO:teuthology.orchestra.run.smithi118.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T02:10:48.196 INFO:teuthology.orchestra.run.smithi118.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T02:10:49.364 INFO:teuthology.orchestra.run.smithi177.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T02:10:49.830 INFO:teuthology.orchestra.run.smithi003.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T02:10:50.033 INFO:teuthology.orchestra.run.smithi177.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T02:10:50.056 INFO:teuthology.orchestra.run.smithi003.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T02:10:50.056 INFO:teuthology.orchestra.run.smithi003.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T02:10:50.066 INFO:teuthology.orchestra.run.smithi177.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T02:10:50.069 INFO:teuthology.orchestra.run.smithi177.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T02:10:50.144 INFO:teuthology.orchestra.run.smithi118.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T02:10:50.146 INFO:teuthology.orchestra.run.smithi118.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T02:10:50.147 INFO:teuthology.orchestra.run.smithi118.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T02:10:50.230 INFO:teuthology.orchestra.run.smithi177.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T02:10:50.298 INFO:teuthology.orchestra.run.smithi195.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-16T02:10:50.717 INFO:teuthology.orchestra.run.smithi118.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T02:10:50.790 INFO:teuthology.orchestra.run.smithi195.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-16T02:10:50.795 INFO:teuthology.orchestra.run.smithi195.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-16T02:10:50.812 INFO:teuthology.orchestra.run.smithi195.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-16T02:10:50.955 INFO:teuthology.orchestra.run.smithi177.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T02:10:51.106 INFO:teuthology.orchestra.run.smithi195.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-16T02:10:51.291 INFO:teuthology.orchestra.run.smithi118.stdout:Fetched 244 MB in 8s (28.9 MB/s) 2024-09-16T02:10:51.454 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T02:10:51.515 INFO:teuthology.orchestra.run.smithi195.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-16T02:10:51.550 INFO:teuthology.orchestra.run.smithi118.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T02:10:51.555 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T02:10:51.628 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:10:51.774 INFO:teuthology.orchestra.run.smithi003.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T02:10:51.776 INFO:teuthology.orchestra.run.smithi003.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T02:10:51.777 INFO:teuthology.orchestra.run.smithi003.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T02:10:51.959 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T02:10:51.986 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T02:10:52.025 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:10:52.193 INFO:teuthology.orchestra.run.smithi003.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T02:10:52.329 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T02:10:52.355 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T02:10:52.395 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:10:52.579 INFO:teuthology.orchestra.run.smithi177.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T02:10:52.580 INFO:teuthology.orchestra.run.smithi177.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T02:10:52.596 INFO:teuthology.orchestra.run.smithi177.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T02:10:52.709 INFO:teuthology.orchestra.run.smithi177.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T02:10:52.748 INFO:teuthology.orchestra.run.smithi177.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T02:10:52.753 INFO:teuthology.orchestra.run.smithi177.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T02:10:52.784 INFO:teuthology.orchestra.run.smithi195.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-16T02:10:52.785 INFO:teuthology.orchestra.run.smithi003.stdout:Fetched 244 MB in 10s (24.6 MB/s) 2024-09-16T02:10:52.785 INFO:teuthology.orchestra.run.smithi195.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-16T02:10:52.802 INFO:teuthology.orchestra.run.smithi195.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-16T02:10:52.808 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T02:10:52.834 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:52.882 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:52.895 INFO:teuthology.orchestra.run.smithi195.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-16T02:10:52.933 INFO:teuthology.orchestra.run.smithi195.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-16T02:10:52.936 INFO:teuthology.orchestra.run.smithi195.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-16T02:10:52.951 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T02:10:53.053 INFO:teuthology.orchestra.run.smithi003.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T02:10:53.059 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T02:10:53.141 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:10:53.353 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T02:10:53.379 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:53.419 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:53.453 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T02:10:53.480 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T02:10:53.511 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:10:53.714 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T02:10:53.741 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:53.781 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:53.847 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T02:10:53.874 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T02:10:53.914 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:10:54.126 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T02:10:54.152 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T02:10:54.201 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:10:54.351 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T02:10:54.378 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:54.426 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:54.487 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librados2. 2024-09-16T02:10:54.513 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:54.545 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:54.754 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T02:10:54.770 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:54.804 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:55.065 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libnbd0. 2024-09-16T02:10:55.091 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T02:10:55.124 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T02:10:55.132 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T02:10:55.150 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:10:55.207 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:10:55.443 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librbd1. 2024-09-16T02:10:55.469 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:55.510 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:55.568 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T02:10:55.595 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T02:10:55.635 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:10:55.646 INFO:teuthology.orchestra.run.smithi195.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T02:10:55.727 INFO:teuthology.orchestra.run.smithi177.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-16T02:10:55.865 INFO:teuthology.orchestra.run.smithi195.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T02:10:55.865 INFO:teuthology.orchestra.run.smithi195.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T02:10:55.896 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T02:10:55.919 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:55.949 INFO:teuthology.orchestra.run.smithi177.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-16T02:10:55.950 INFO:teuthology.orchestra.run.smithi177.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-16T02:10:55.954 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:56.007 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librados2. 2024-09-16T02:10:56.034 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:56.196 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:56.215 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rados. 2024-09-16T02:10:56.230 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:56.274 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:56.594 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T02:10:56.620 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:10:56.658 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libnbd0. 2024-09-16T02:10:56.660 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:56.684 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T02:10:56.724 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T02:10:56.929 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T02:10:56.944 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:56.979 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:57.035 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librbd1. 2024-09-16T02:10:57.062 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:57.102 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:57.232 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T02:10:57.256 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:10:57.307 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:57.505 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T02:10:57.531 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:57.563 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:57.820 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T02:10:57.824 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rados. 2024-09-16T02:10:57.839 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:57.846 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T02:10:57.877 INFO:teuthology.orchestra.run.smithi195.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T02:10:57.880 INFO:teuthology.orchestra.run.smithi195.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T02:10:57.881 INFO:teuthology.orchestra.run.smithi195.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T02:10:57.882 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:57.886 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:10:57.984 INFO:teuthology.orchestra.run.smithi177.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-16T02:10:57.987 INFO:teuthology.orchestra.run.smithi177.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-16T02:10:57.988 INFO:teuthology.orchestra.run.smithi177.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-16T02:10:58.185 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T02:10:58.206 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T02:10:58.211 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:10:58.232 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T02:10:58.251 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:58.272 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T02:10:58.363 INFO:teuthology.orchestra.run.smithi195.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T02:10:58.487 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T02:10:58.503 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:58.508 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T02:10:58.515 INFO:teuthology.orchestra.run.smithi177.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-16T02:10:58.524 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:58.545 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:58.558 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:58.823 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T02:10:58.850 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:10:58.881 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:58.895 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T02:10:58.922 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T02:10:58.932 INFO:teuthology.orchestra.run.smithi195.stdout:Fetched 244 MB in 16s (15.3 MB/s) 2024-09-16T02:10:58.961 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:10:59.094 INFO:teuthology.orchestra.run.smithi177.stdout:Fetched 244 MB in 16s (15.1 MB/s) 2024-09-16T02:10:59.105 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T02:10:59.151 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T02:10:59.178 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T02:10:59.206 INFO:teuthology.orchestra.run.smithi195.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T02:10:59.209 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:10:59.211 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T02:10:59.248 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T02:10:59.253 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T02:10:59.263 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T02:10:59.295 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:10:59.306 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:10:59.353 INFO:teuthology.orchestra.run.smithi177.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T02:10:59.359 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T02:10:59.443 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:10:59.579 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T02:10:59.606 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T02:10:59.609 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T02:10:59.625 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T02:10:59.632 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T02:10:59.645 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T02:10:59.658 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T02:10:59.659 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:10:59.698 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:10:59.772 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T02:10:59.798 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T02:10:59.830 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:10:59.898 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T02:10:59.915 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:10:59.956 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:10:59.971 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua5.1. 2024-09-16T02:10:59.997 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T02:11:00.018 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T02:11:00.037 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:00.044 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T02:11:00.085 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:11:00.309 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T02:11:00.316 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-any. 2024-09-16T02:11:00.333 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T02:11:00.336 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T02:11:00.342 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T02:11:00.360 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T02:11:00.375 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:11:00.391 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T02:11:00.399 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:11:00.538 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T02:11:00.565 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:00.611 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T02:11:00.613 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:00.627 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T02:11:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package zip. 2024-09-16T02:11:00.661 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:11:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T02:11:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T02:11:00.844 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T02:11:00.871 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:00.919 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:00.931 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T02:11:00.947 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T02:11:00.949 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T02:11:00.965 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:00.989 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:11:00.999 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:01.055 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package unzip. 2024-09-16T02:11:01.082 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T02:11:01.122 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:01.272 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T02:11:01.299 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:01.330 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:01.361 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T02:11:01.387 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:01.428 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:01.442 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua5.1. 2024-09-16T02:11:01.469 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T02:11:01.501 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package luarocks. 2024-09-16T02:11:01.509 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:01.527 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T02:11:01.567 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:11:01.659 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T02:11:01.685 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T02:11:01.725 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:11:01.812 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-any. 2024-09-16T02:11:01.822 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T02:11:01.839 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T02:11:01.848 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T02:11:01.870 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T02:11:01.889 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:11:01.901 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package librgw2. 2024-09-16T02:11:01.927 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:01.967 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:02.073 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package zip. 2024-09-16T02:11:02.078 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T02:11:02.089 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T02:11:02.105 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T02:11:02.131 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T02:11:02.144 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:11:02.208 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librados2. 2024-09-16T02:11:02.235 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:02.284 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:02.442 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package unzip. 2024-09-16T02:11:02.447 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librados2. 2024-09-16T02:11:02.470 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T02:11:02.474 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:02.506 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:02.509 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:02.587 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T02:11:02.602 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:02.645 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:02.803 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libnbd0. 2024-09-16T02:11:02.829 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T02:11:02.870 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T02:11:02.905 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package luarocks. 2024-09-16T02:11:02.931 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T02:11:02.971 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:11:03.000 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libnbd0. 2024-09-16T02:11:03.015 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T02:11:03.026 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-16T02:11:03.041 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T02:11:03.066 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-16T02:11:03.081 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:03.156 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librbd1. 2024-09-16T02:11:03.182 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.222 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.360 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T02:11:03.370 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package librgw2. 2024-09-16T02:11:03.378 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librbd1. 2024-09-16T02:11:03.386 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.398 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.404 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.426 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.437 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.444 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.642 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T02:11:03.668 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.709 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.830 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-common. 2024-09-16T02:11:03.857 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.872 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T02:11:03.888 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:03.896 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:03.930 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.011 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rados. 2024-09-16T02:11:04.027 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.064 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T02:11:04.069 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.081 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.123 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.174 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rados. 2024-09-16T02:11:04.190 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.233 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.414 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T02:11:04.440 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:04.459 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T02:11:04.480 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.486 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T02:11:04.525 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:04.552 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T02:11:04.578 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:04.618 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.716 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T02:11:04.731 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.766 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.795 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T02:11:04.811 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.845 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:04.871 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T02:11:04.898 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:04.938 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:05.077 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T02:11:05.104 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:05.144 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:05.207 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T02:11:05.234 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:05.265 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-common. 2024-09-16T02:11:05.273 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:05.292 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:05.331 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:05.436 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-base. 2024-09-16T02:11:05.462 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:05.497 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T02:11:05.504 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:05.523 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T02:11:05.564 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:11:05.585 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T02:11:05.611 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T02:11:05.653 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:11:05.883 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T02:11:05.910 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T02:11:05.949 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T02:11:05.971 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T02:11:05.997 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T02:11:06.037 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T02:11:06.082 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T02:11:06.108 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T02:11:06.148 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:06.202 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T02:11:06.218 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:06.252 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:06.298 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T02:11:06.313 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:06.356 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:06.418 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T02:11:06.444 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T02:11:06.476 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:11:06.651 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T02:11:06.653 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T02:11:06.677 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T02:11:06.680 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T02:11:06.709 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:11:06.746 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T02:11:06.761 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T02:11:06.763 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:11:06.796 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:06.903 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-base. 2024-09-16T02:11:06.930 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:06.980 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:06.987 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T02:11:07.009 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T02:11:07.049 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-16T02:11:07.053 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:11:07.064 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-16T02:11:07.091 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T02:11:07.107 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:11:07.117 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T02:11:07.157 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:11:07.358 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T02:11:07.385 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T02:11:07.402 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-16T02:11:07.417 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-16T02:11:07.424 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:11:07.452 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:11:07.477 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T02:11:07.499 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T02:11:07.503 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T02:11:07.515 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T02:11:07.544 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:11:07.549 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:07.728 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua5.1. 2024-09-16T02:11:07.754 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T02:11:07.772 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua5.1. 2024-09-16T02:11:07.786 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:07.788 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T02:11:07.798 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-16T02:11:07.804 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T02:11:07.835 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T02:11:07.838 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T02:11:07.838 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:07.862 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T02:11:07.905 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:11:08.109 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-any. 2024-09-16T02:11:08.132 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T02:11:08.158 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-any. 2024-09-16T02:11:08.172 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T02:11:08.184 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-16T02:11:08.213 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T02:11:08.225 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-16T02:11:08.240 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T02:11:08.280 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:08.308 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-portend. 2024-09-16T02:11:08.335 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T02:11:08.375 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T02:11:08.400 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package zip. 2024-09-16T02:11:08.426 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T02:11:08.462 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package zip. 2024-09-16T02:11:08.466 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T02:11:08.489 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-16T02:11:08.529 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking zip (3.0-12build2) ... 2024-09-16T02:11:08.582 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T02:11:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T02:11:08.649 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:11:08.695 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T02:11:08.721 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T02:11:08.761 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T02:11:08.769 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package unzip. 2024-09-16T02:11:08.784 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T02:11:08.819 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:08.918 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T02:11:08.924 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package unzip. 2024-09-16T02:11:08.935 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T02:11:08.950 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-16T02:11:08.968 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:11:08.999 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:09.064 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T02:11:09.091 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T02:11:09.131 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:11:09.181 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package luarocks. 2024-09-16T02:11:09.207 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T02:11:09.256 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:11:09.287 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T02:11:09.298 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T02:11:09.338 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T02:11:09.419 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package luarocks. 2024-09-16T02:11:09.446 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-16T02:11:09.468 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T02:11:09.495 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T02:11:09.495 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:11:09.535 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T02:11:09.580 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package librgw2. 2024-09-16T02:11:09.607 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:09.638 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:09.640 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-portend. 2024-09-16T02:11:09.668 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T02:11:09.707 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T02:11:09.796 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T02:11:09.823 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T02:11:09.863 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T02:11:09.885 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package librgw2. 2024-09-16T02:11:09.912 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:09.952 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:09.993 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T02:11:10.021 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T02:11:10.060 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T02:11:10.174 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T02:11:10.201 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T02:11:10.241 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:10.249 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T02:11:10.276 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:10.338 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T02:11:10.365 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T02:11:10.404 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:11:10.474 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:10.553 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-mako. 2024-09-16T02:11:10.564 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T02:11:10.578 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:10.579 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T02:11:10.619 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:11:10.621 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:10.733 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T02:11:10.760 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T02:11:10.799 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T02:11:10.827 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T02:11:10.853 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T02:11:10.893 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:10.939 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T02:11:10.954 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T02:11:10.989 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:11:11.024 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T02:11:11.050 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T02:11:11.069 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T02:11:11.096 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T02:11:11.099 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:11.135 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T02:11:11.205 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T02:11:11.231 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:11.250 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T02:11:11.263 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:11.277 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T02:11:11.317 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:11:11.438 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T02:11:11.443 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T02:11:11.465 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T02:11:11.469 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:11.505 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:11.510 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:11.595 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webob. 2024-09-16T02:11:11.621 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T02:11:11.661 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:11:11.666 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-common. 2024-09-16T02:11:11.693 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:11.733 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:11.808 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-mako. 2024-09-16T02:11:11.835 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T02:11:11.874 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:11:11.998 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T02:11:12.024 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T02:11:12.055 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-common. 2024-09-16T02:11:12.068 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:12.081 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:12.121 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:12.202 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T02:11:12.230 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T02:11:12.269 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:11:12.385 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T02:11:12.411 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T02:11:12.451 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:11:12.539 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T02:11:12.566 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T02:11:12.605 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:11:12.704 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-paste. 2024-09-16T02:11:12.731 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T02:11:12.771 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:11:12.900 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webob. 2024-09-16T02:11:12.926 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T02:11:12.966 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:11:13.099 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T02:11:13.126 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T02:11:13.166 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:11:13.171 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-base. 2024-09-16T02:11:13.197 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:13.239 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:13.311 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T02:11:13.338 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T02:11:13.380 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:13.452 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T02:11:13.479 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T02:11:13.519 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:11:13.619 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-base. 2024-09-16T02:11:13.645 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:13.694 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:13.830 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T02:11:13.857 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T02:11:13.897 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T02:11:14.200 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T02:11:14.226 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T02:11:14.266 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:11:14.373 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T02:11:14.401 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T02:11:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:11:14.584 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T02:11:14.611 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T02:11:14.647 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T02:11:14.650 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:14.674 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T02:11:14.713 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T02:11:14.750 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-paste. 2024-09-16T02:11:14.777 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T02:11:14.808 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:11:14.953 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T02:11:14.965 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T02:11:14.979 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T02:11:14.991 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T02:11:15.000 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-idna. 2024-09-16T02:11:15.020 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:11:15.026 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T02:11:15.031 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:15.058 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:11:15.103 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T02:11:15.119 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T02:11:15.161 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:11:15.301 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T02:11:15.316 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T02:11:15.348 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T02:11:15.351 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:11:15.374 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T02:11:15.403 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T02:11:15.423 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:15.429 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T02:11:15.469 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:11:15.472 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T02:11:15.500 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T02:11:15.539 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:11:15.662 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T02:11:15.689 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T02:11:15.729 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:15.783 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T02:11:15.800 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T02:11:15.833 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T02:11:15.934 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T02:11:15.960 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T02:11:16.000 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:11:16.031 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T02:11:16.058 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T02:11:16.106 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:11:16.128 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T02:11:16.155 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T02:11:16.194 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:11:16.253 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T02:11:16.279 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T02:11:16.319 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:11:16.409 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T02:11:16.435 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T02:11:16.475 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:11:16.558 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T02:11:16.585 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T02:11:16.616 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests. 2024-09-16T02:11:16.622 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T02:11:16.624 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T02:11:16.642 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T02:11:16.648 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T02:11:16.682 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:11:16.689 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T02:11:16.737 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T02:11:16.763 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T02:11:16.795 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T02:11:17.009 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-portend. 2024-09-16T02:11:17.028 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T02:11:17.035 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T02:11:17.054 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T02:11:17.075 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T02:11:17.093 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:11:17.169 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-idna. 2024-09-16T02:11:17.196 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T02:11:17.235 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:11:17.344 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T02:11:17.359 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T02:11:17.365 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-portend. 2024-09-16T02:11:17.391 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-16T02:11:17.394 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T02:11:17.422 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T02:11:17.431 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T02:11:17.448 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:17.488 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:17.555 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T02:11:17.582 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T02:11:17.615 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:11:17.672 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T02:11:17.698 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T02:11:17.734 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T02:11:17.738 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:11:17.760 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T02:11:17.800 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T02:11:17.852 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T02:11:17.867 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-requests. 2024-09-16T02:11:17.879 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:17.884 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T02:11:17.918 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:11:17.918 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:18.075 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T02:11:18.087 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T02:11:18.091 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T02:11:18.113 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T02:11:18.125 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T02:11:18.145 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:11:18.212 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T02:11:18.230 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T02:11:18.239 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T02:11:18.256 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T02:11:18.296 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:11:18.296 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:18.403 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T02:11:18.430 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T02:11:18.461 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T02:11:18.532 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T02:11:18.549 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T02:11:18.559 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T02:11:18.565 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:18.599 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:18.599 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T02:11:18.607 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T02:11:18.626 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T02:11:18.666 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T02:11:18.756 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T02:11:18.782 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T02:11:18.822 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:18.945 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T02:11:18.973 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:18.978 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T02:11:18.985 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T02:11:19.004 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:19.011 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T02:11:19.012 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:19.044 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:19.060 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T02:11:19.125 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-mako. 2024-09-16T02:11:19.152 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T02:11:19.192 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:11:19.264 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T02:11:19.280 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T02:11:19.314 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:19.373 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T02:11:19.399 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:19.404 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T02:11:19.431 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:19.431 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T02:11:19.471 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:19.553 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T02:11:19.580 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T02:11:19.592 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T02:11:19.620 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:11:19.620 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T02:11:19.658 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T02:11:19.791 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-mako. 2024-09-16T02:11:19.817 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T02:11:19.857 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:11:19.881 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T02:11:19.907 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T02:11:19.939 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:11:19.986 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T02:11:20.014 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:20.053 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:20.068 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T02:11:20.095 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:20.134 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:20.152 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T02:11:20.178 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T02:11:20.208 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webob. 2024-09-16T02:11:20.219 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:11:20.235 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T02:11:20.275 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:11:20.448 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T02:11:20.475 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:20.497 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T02:11:20.514 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:20.523 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T02:11:20.563 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:11:20.619 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T02:11:20.646 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T02:11:20.689 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:20.866 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-webob. 2024-09-16T02:11:20.892 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T02:11:20.932 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:11:20.997 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T02:11:21.024 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T02:11:21.072 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:11:21.085 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T02:11:21.112 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:21.142 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:21.285 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T02:11:21.288 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph. 2024-09-16T02:11:21.312 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T02:11:21.314 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:21.354 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:21.355 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:21.383 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-paste. 2024-09-16T02:11:21.410 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T02:11:21.450 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:11:21.632 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T02:11:21.648 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:21.655 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T02:11:21.681 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T02:11:21.690 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:21.721 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:11:21.778 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T02:11:21.793 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T02:11:21.836 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:11:21.974 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-paste. 2024-09-16T02:11:21.989 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T02:11:22.024 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:11:22.086 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T02:11:22.112 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:22.122 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T02:11:22.149 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T02:11:22.152 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:22.187 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph. 2024-09-16T02:11:22.188 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:11:22.204 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:22.246 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:22.344 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T02:11:22.359 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T02:11:22.402 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:11:22.475 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T02:11:22.502 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T02:11:22.541 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T02:11:22.565 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T02:11:22.572 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package cephadm. 2024-09-16T02:11:22.592 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:22.599 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:22.631 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:22.638 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:22.680 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T02:11:22.707 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T02:11:22.738 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:11:22.794 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T02:11:22.821 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T02:11:22.852 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:11:22.950 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T02:11:22.968 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T02:11:22.976 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T02:11:22.984 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:23.016 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T02:11:23.017 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:23.041 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T02:11:23.068 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T02:11:23.108 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T02:11:23.165 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T02:11:23.181 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T02:11:23.223 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T02:11:23.286 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T02:11:23.301 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:23.336 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:23.429 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package cephadm. 2024-09-16T02:11:23.444 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T02:11:23.456 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:23.470 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T02:11:23.511 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:11:23.560 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-idna. 2024-09-16T02:11:23.586 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T02:11:23.612 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:23.626 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:11:23.874 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T02:11:23.900 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T02:11:23.923 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T02:11:23.937 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T02:11:23.940 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T02:11:23.950 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T02:11:23.964 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T02:11:23.989 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T02:11:24.004 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:11:24.268 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-idna. 2024-09-16T02:11:24.292 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T02:11:24.294 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T02:11:24.308 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:24.334 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:11:24.341 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests. 2024-09-16T02:11:24.350 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:24.367 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T02:11:24.383 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-babel. 2024-09-16T02:11:24.407 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:11:24.410 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:24.449 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:24.613 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T02:11:24.628 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T02:11:24.662 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:11:24.726 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T02:11:24.753 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T02:11:24.787 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T02:11:24.793 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:11:24.813 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T02:11:24.853 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:11:24.924 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-requests. 2024-09-16T02:11:24.950 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T02:11:24.991 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:11:25.096 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T02:11:25.111 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:25.154 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:25.214 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T02:11:25.241 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:25.281 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:25.293 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T02:11:25.320 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T02:11:25.339 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-babel. 2024-09-16T02:11:25.360 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:11:25.366 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:25.405 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:25.509 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T02:11:25.536 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:25.575 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:25.601 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T02:11:25.628 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T02:11:25.630 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T02:11:25.645 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:25.676 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:25.688 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:25.733 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T02:11:25.761 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T02:11:25.800 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:11:25.887 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T02:11:25.913 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T02:11:25.953 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:25.954 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-routes. 2024-09-16T02:11:25.981 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T02:11:26.020 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:26.076 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T02:11:26.103 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:26.143 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:26.144 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T02:11:26.172 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:26.211 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:26.374 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T02:11:26.401 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:26.431 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T02:11:26.437 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T02:11:26.440 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:26.457 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T02:11:26.464 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T02:11:26.495 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:26.497 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T02:11:26.564 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T02:11:26.592 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T02:11:26.630 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:26.776 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T02:11:26.782 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T02:11:26.802 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:26.808 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T02:11:26.833 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:26.865 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T02:11:26.917 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-routes. 2024-09-16T02:11:26.933 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T02:11:26.975 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:27.210 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T02:11:27.211 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T02:11:27.236 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:27.238 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:27.276 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:27.278 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:27.487 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T02:11:27.514 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:27.553 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:27.688 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T02:11:27.714 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:27.754 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:27.898 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T02:11:27.925 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:27.964 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:28.324 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T02:11:28.351 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:28.391 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:29.043 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph. 2024-09-16T02:11:29.058 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:29.101 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:29.370 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T02:11:29.385 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:29.420 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:29.435 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph. 2024-09-16T02:11:29.451 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:29.494 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:29.530 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T02:11:29.547 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T02:11:29.580 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:29.790 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T02:11:29.816 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:29.822 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T02:11:29.848 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:29.856 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:29.905 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:30.017 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T02:11:30.044 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T02:11:30.083 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:30.225 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T02:11:30.250 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T02:11:30.251 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:30.267 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package cephadm. 2024-09-16T02:11:30.280 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T02:11:30.282 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:30.294 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:30.324 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:30.334 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:30.489 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T02:11:30.516 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T02:11:30.555 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:30.661 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T02:11:30.669 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package cephadm. 2024-09-16T02:11:30.688 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T02:11:30.695 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:30.727 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:30.728 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T02:11:30.828 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T02:11:30.856 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T02:11:30.858 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T02:11:30.886 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T02:11:30.894 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:31.010 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:11:31.047 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T02:11:31.056 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T02:11:31.073 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T02:11:31.082 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:31.113 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T02:11:31.122 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:31.281 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T02:11:31.309 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T02:11:31.347 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:31.408 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T02:11:31.436 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:31.474 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:31.650 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T02:11:31.678 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T02:11:31.827 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T02:11:31.828 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:11:31.844 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:31.885 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:32.110 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-babel. 2024-09-16T02:11:32.126 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:32.160 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:32.446 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T02:11:32.473 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T02:11:32.504 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:11:32.586 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T02:11:32.616 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:32.653 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:32.654 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-babel. 2024-09-16T02:11:32.681 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T02:11:32.721 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:32.763 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T02:11:32.779 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T02:11:32.790 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T02:11:32.806 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:32.821 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:32.840 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:33.024 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T02:11:33.050 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T02:11:33.090 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:11:33.124 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T02:11:33.152 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T02:11:33.153 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T02:11:33.167 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T02:11:33.190 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:11:33.201 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:33.410 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T02:11:33.436 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:33.476 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:33.496 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-routes. 2024-09-16T02:11:33.513 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T02:11:33.519 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T02:11:33.547 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T02:11:33.554 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:33.585 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:11:33.667 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T02:11:33.696 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T02:11:33.781 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:33.830 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T02:11:33.857 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T02:11:33.896 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:33.941 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T02:11:33.968 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:34.007 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:34.051 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T02:11:34.079 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T02:11:34.109 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:11:34.122 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T02:11:34.150 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T02:11:34.188 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T02:11:34.233 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-routes. 2024-09-16T02:11:34.260 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T02:11:34.299 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:34.379 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T02:11:34.407 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T02:11:34.437 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:11:34.509 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T02:11:34.537 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T02:11:34.575 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T02:11:34.644 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T02:11:34.671 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:34.710 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:34.723 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T02:11:34.752 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T02:11:34.789 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T02:11:34.912 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T02:11:34.939 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T02:11:34.978 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:11:35.110 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T02:11:35.139 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T02:11:35.177 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T02:11:35.331 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T02:11:35.359 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T02:11:35.397 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T02:11:35.522 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T02:11:35.550 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T02:11:35.588 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:11:35.709 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T02:11:35.737 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T02:11:35.775 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:11:35.891 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T02:11:35.920 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T02:11:35.958 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T02:11:36.269 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T02:11:36.298 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T02:11:36.335 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:11:36.414 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T02:11:36.443 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:36.480 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:36.767 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T02:11:36.795 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T02:11:36.825 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:11:36.978 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T02:11:36.995 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T02:11:37.028 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:37.041 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T02:11:37.059 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:37.099 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:37.144 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T02:11:37.172 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:37.211 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:11:37.374 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T02:11:37.392 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T02:11:37.410 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T02:11:37.432 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:37.439 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T02:11:37.472 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package jq. 2024-09-16T02:11:37.477 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:11:37.481 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T02:11:37.500 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:37.509 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T02:11:37.538 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T02:11:37.548 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:37.791 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package socat. 2024-09-16T02:11:37.807 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T02:11:37.849 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:11:37.927 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T02:11:37.954 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T02:11:37.969 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T02:11:37.971 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T02:11:37.985 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T02:11:37.993 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:38.000 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:38.019 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:38.029 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:11:38.144 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T02:11:38.160 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T02:11:38.194 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:11:38.245 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T02:11:38.262 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T02:11:38.265 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package jq. 2024-09-16T02:11:38.283 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:38.324 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T02:11:38.380 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:11:38.389 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T02:11:38.416 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T02:11:38.455 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:38.539 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-test. 2024-09-16T02:11:38.556 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:38.585 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package socat. 2024-09-16T02:11:38.598 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:38.602 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T02:11:38.643 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:11:38.733 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T02:11:38.760 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T02:11:38.893 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:11:38.988 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T02:11:39.017 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T02:11:39.054 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:11:39.090 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T02:11:39.118 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:39.156 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:39.425 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-test. 2024-09-16T02:11:39.454 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:39.492 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:39.594 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T02:11:39.622 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:39.661 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:40.151 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T02:11:40.179 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T02:11:40.217 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:40.520 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T02:11:40.548 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T02:11:40.587 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:11:40.631 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T02:11:40.660 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T02:11:40.697 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:40.923 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T02:11:40.951 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T02:11:40.958 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T02:11:40.986 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T02:11:40.989 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:11:41.016 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:11:41.301 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T02:11:41.329 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T02:11:41.353 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T02:11:41.367 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T02:11:41.381 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T02:11:41.420 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:11:41.569 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T02:11:41.597 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:41.635 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:41.654 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T02:11:41.670 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T02:11:41.704 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T02:11:41.706 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T02:11:41.734 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-16T02:11:41.763 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T02:11:41.773 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T02:11:41.792 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:41.829 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:41.930 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T02:11:41.947 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:41.988 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:42.008 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T02:11:42.036 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T02:11:42.068 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T02:11:42.074 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:11:42.096 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T02:11:42.135 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T02:11:42.142 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T02:11:42.171 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:42.208 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:42.300 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T02:11:42.328 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T02:11:42.366 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:42.385 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T02:11:42.413 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T02:11:42.452 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T02:11:42.463 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T02:11:42.486 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T02:11:42.491 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T02:11:42.515 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T02:11:42.538 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:11:42.585 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:42.696 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T02:11:42.724 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T02:11:42.762 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:11:42.763 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T02:11:42.791 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T02:11:42.830 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:11:42.849 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T02:11:42.877 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T02:11:42.915 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T02:11:42.915 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T02:11:42.944 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T02:11:42.981 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:11:43.066 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T02:11:43.093 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T02:11:43.124 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:43.227 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T02:11:43.255 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T02:11:43.294 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:11:43.311 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T02:11:43.339 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T02:11:43.377 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:43.469 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T02:11:43.497 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:43.504 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package pkg-config. 2024-09-16T02:11:43.526 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:43.531 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T02:11:43.570 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:11:43.732 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package pkg-config. 2024-09-16T02:11:43.761 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T02:11:43.790 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:11:43.815 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T02:11:43.843 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T02:11:43.844 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T02:11:43.873 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T02:11:43.898 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:43.912 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:11:43.949 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T02:11:43.966 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:44.016 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:44.018 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T02:11:44.035 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T02:11:44.076 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:44.232 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T02:11:44.260 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:44.267 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T02:11:44.295 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T02:11:44.307 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:11:44.334 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:44.419 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T02:11:44.447 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T02:11:44.493 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:11:44.496 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T02:11:44.525 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T02:11:44.535 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package jq. 2024-09-16T02:11:44.551 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:44.562 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:44.593 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T02:11:44.637 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T02:11:44.664 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T02:11:44.703 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T02:11:44.845 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package socat. 2024-09-16T02:11:44.862 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T02:11:44.863 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T02:11:44.865 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T02:11:44.892 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:44.894 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T02:11:44.903 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:11:44.930 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:11:44.931 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T02:11:45.032 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T02:11:45.059 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T02:11:45.098 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:11:45.216 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package jq. 2024-09-16T02:11:45.223 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T02:11:45.240 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T02:11:45.244 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T02:11:45.260 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T02:11:45.281 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:11:45.289 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T02:11:45.290 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T02:11:45.326 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:11:45.409 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T02:11:45.437 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T02:11:45.475 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:45.594 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package socat. 2024-09-16T02:11:45.621 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T02:11:45.638 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T02:11:45.653 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-test. 2024-09-16T02:11:45.667 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T02:11:45.668 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:11:45.670 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:45.696 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:45.711 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:45.796 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-py. 2024-09-16T02:11:45.823 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T02:11:45.861 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T02:11:46.030 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T02:11:46.057 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-py. 2024-09-16T02:11:46.058 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T02:11:46.086 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T02:11:46.096 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:11:46.148 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T02:11:46.165 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T02:11:46.193 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T02:11:46.231 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:11:46.460 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T02:11:46.489 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T02:11:46.492 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-test. 2024-09-16T02:11:46.509 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:46.528 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:11:46.559 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:46.702 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T02:11:46.730 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T02:11:46.768 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:46.956 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T02:11:46.985 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T02:11:47.022 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:47.113 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-toml. 2024-09-16T02:11:47.141 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T02:11:47.179 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T02:11:47.317 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-toml. 2024-09-16T02:11:47.346 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T02:11:47.383 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T02:11:47.474 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T02:11:47.501 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T02:11:47.531 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:11:47.686 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T02:11:47.715 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T02:11:47.752 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:11:47.868 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T02:11:47.896 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T02:11:47.934 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:47.973 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T02:11:48.002 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:48.039 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:48.114 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T02:11:48.143 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T02:11:48.170 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T02:11:48.180 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:48.186 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T02:11:48.237 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:11:48.329 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T02:11:48.345 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:48.387 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:48.458 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T02:11:48.487 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T02:11:48.525 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:11:48.656 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T02:11:48.684 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T02:11:48.714 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:48.924 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T02:11:48.951 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-16T02:11:48.966 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package radosgw. 2024-09-16T02:11:48.981 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:48.995 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:49.019 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T02:11:49.032 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:49.047 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T02:11:49.085 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:11:49.195 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package radosgw. 2024-09-16T02:11:49.225 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:49.253 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:49.277 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T02:11:49.293 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:49.327 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:49.456 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T02:11:49.483 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T02:11:49.522 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:49.588 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-16T02:11:49.604 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-16T02:11:49.638 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:49.910 INFO:teuthology.orchestra.run.smithi118.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T02:11:49.939 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:49.942 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-16T02:11:49.952 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package pkg-config. 2024-09-16T02:11:49.970 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-16T02:11:49.976 INFO:teuthology.orchestra.run.smithi118.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:49.980 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T02:11:50.000 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:11:50.018 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:11:50.263 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T02:11:50.273 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:50.291 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T02:11:50.329 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:50.348 INFO:teuthology.orchestra.run.smithi003.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T02:11:50.377 INFO:teuthology.orchestra.run.smithi003.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:50.380 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T02:11:50.407 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T02:11:50.414 INFO:teuthology.orchestra.run.smithi003.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:50.446 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:50.515 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:11:50.616 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:50.711 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:50.716 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T02:11:50.744 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T02:11:50.782 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:50.868 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package pkg-config. 2024-09-16T02:11:50.896 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-16T02:11:50.898 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T02:11:50.934 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:11:50.961 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:11:51.085 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T02:11:51.095 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:51.113 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T02:11:51.151 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T02:11:51.187 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T02:11:51.214 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T02:11:51.216 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T02:11:51.254 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:51.364 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T02:11:51.472 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T02:11:51.500 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T02:11:51.538 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:11:51.644 INFO:teuthology.orchestra.run.smithi118.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T02:11:51.674 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T02:11:51.702 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T02:11:51.709 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:51.711 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T02:11:51.740 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:51.866 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T02:11:51.894 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T02:11:51.932 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:52.068 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T02:11:52.096 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T02:11:52.134 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T02:11:52.141 INFO:teuthology.orchestra.run.smithi003.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T02:11:52.239 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:52.259 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user cephadm....done 2024-09-16T02:11:52.294 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-py. 2024-09-16T02:11:52.322 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T02:11:52.360 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T02:11:52.382 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:52.455 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T02:11:52.483 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T02:11:52.521 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:11:52.688 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T02:11:52.699 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:52.716 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T02:11:52.755 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:11:52.816 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T02:11:52.839 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user cephadm....done 2024-09-16T02:11:52.844 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T02:11:52.882 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:52.993 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:53.119 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:53.175 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T02:11:53.203 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T02:11:53.233 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:53.243 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-py. 2024-09-16T02:11:53.272 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-16T02:11:53.318 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T02:11:53.386 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:53.497 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:53.553 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-toml. 2024-09-16T02:11:53.581 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T02:11:53.619 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T02:11:53.654 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:53.663 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T02:11:53.692 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T02:11:53.730 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:11:53.780 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T02:11:53.922 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T02:11:53.950 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T02:11:53.965 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:53.988 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:11:54.098 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:54.143 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T02:11:54.171 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T02:11:54.209 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:54.225 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:54.334 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T02:11:54.351 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:54.362 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T02:11:54.400 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:54.458 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:54.537 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-toml. 2024-09-16T02:11:54.565 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-16T02:11:54.603 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T02:11:54.610 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T02:11:54.661 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T02:11:54.678 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T02:11:54.719 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:11:54.793 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T02:11:54.878 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:11:54.889 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T02:11:54.917 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T02:11:54.955 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:11:55.004 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T02:11:55.069 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:55.309 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T02:11:55.320 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:55.322 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:11:55.338 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T02:11:55.412 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:55.465 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package radosgw. 2024-09-16T02:11:55.493 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:55.531 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:55.571 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:55.670 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T02:11:55.691 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T02:11:55.699 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T02:11:55.736 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:11:55.814 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:55.942 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T02:11:56.140 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:56.218 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:11:56.399 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:56.417 INFO:teuthology.orchestra.run.smithi177.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T02:11:56.434 INFO:teuthology.orchestra.run.smithi177.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:56.453 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:11:56.465 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package radosgw. 2024-09-16T02:11:56.475 INFO:teuthology.orchestra.run.smithi177.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:56.494 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:56.525 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T02:11:56.532 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:56.651 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:56.777 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:11:56.779 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:11:56.781 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:56.912 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:57.038 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:11:57.047 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:11:57.164 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:57.171 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:57.289 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:57.408 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T02:11:57.435 INFO:teuthology.orchestra.run.smithi195.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T02:11:57.451 INFO:teuthology.orchestra.run.smithi195.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-16T02:11:57.493 INFO:teuthology.orchestra.run.smithi195.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:57.532 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:11:57.629 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T02:11:57.642 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T02:11:57.660 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T02:11:57.710 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T02:11:57.724 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:11:57.767 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:11:57.806 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T02:11:57.850 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:57.893 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:11:58.019 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:11:58.048 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T02:11:58.136 INFO:teuthology.orchestra.run.smithi177.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T02:11:58.208 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T02:11:58.226 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:58.241 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:58.279 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:11:58.369 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T02:11:58.476 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T02:11:58.636 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:58.661 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T02:11:58.693 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T02:11:58.757 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:11:58.777 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T02:11:58.783 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user cephadm....done 2024-09-16T02:11:58.858 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T02:11:58.896 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:58.973 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:11:59.180 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T02:11:59.210 INFO:teuthology.orchestra.run.smithi195.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T02:11:59.243 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:11:59.285 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:11:59.290 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:11:59.369 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T02:11:59.517 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:11:59.653 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:11:59.751 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:11:59.877 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:11:59.917 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:11:59.921 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:11:59.959 INFO:teuthology.orchestra.run.smithi195.stdout:Adding system user cephadm....done 2024-09-16T02:12:00.116 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T02:12:00.136 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T02:12:00.176 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:12:00.230 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T02:12:00.302 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:12:00.331 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:12:00.409 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T02:12:00.428 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:12:00.429 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:12:00.556 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T02:12:00.660 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T02:12:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T02:12:00.761 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T02:12:00.849 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:12:01.002 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:12:01.003 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T02:12:01.113 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T02:12:01.193 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T02:12:01.261 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:12:01.271 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T02:12:01.364 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:12:01.387 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:12:01.397 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T02:12:01.435 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T02:12:01.513 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:12:01.703 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:12:01.724 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T02:12:01.732 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:12:01.806 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T02:12:01.938 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:12:02.025 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:12:02.093 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T02:12:02.174 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T02:12:02.189 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:12:02.328 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T02:12:02.368 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:12:02.408 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:12:02.431 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:12:02.595 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-16T02:12:02.619 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:12:02.741 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:02.754 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:02.793 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:12:02.897 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T02:12:02.985 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:12:03.046 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:12:03.086 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:12:03.093 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T02:12:03.165 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T02:12:03.219 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:12:03.362 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:12:03.381 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:12:03.441 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T02:12:03.445 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:12:03.487 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:12:03.697 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:12:03.747 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:12:03.806 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:03.959 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:04.099 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:12:04.130 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T02:12:04.162 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T02:12:04.216 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:12:04.260 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T02:12:04.359 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T02:12:04.386 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-16T02:12:04.417 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:12:04.512 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-16T02:12:04.537 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:12:04.646 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-16T02:12:04.719 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:04.797 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:04.797 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T02:12:04.845 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T02:12:04.914 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T02:12:05.081 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-16T02:12:05.121 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:12:05.165 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:12:05.299 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:12:05.388 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:12:05.434 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:12:05.460 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-16T02:12:05.496 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-16T02:12:05.543 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T02:12:05.644 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T02:12:05.681 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T02:12:05.698 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:12:05.735 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:12:05.782 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:12:05.924 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:06.004 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:06.019 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T02:12:06.058 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T02:12:06.146 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T02:12:06.279 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:06.335 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:12:06.405 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:12:06.409 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:12:06.456 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T02:12:06.461 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:12:06.587 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:12:06.660 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:12:06.689 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:12:06.757 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T02:12:06.786 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:12:06.888 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:12:06.912 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:12:06.924 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T02:12:07.092 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up zip (3.0-12build2) ... 2024-09-16T02:12:07.147 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T02:12:07.191 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T02:12:07.201 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T02:12:07.221 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T02:12:07.382 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:07.442 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:12:07.508 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:12:07.598 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T02:12:07.701 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T02:12:07.818 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:12:07.836 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T02:12:07.840 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:12:07.969 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:12:08.077 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T02:12:08.087 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T02:12:08.104 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T02:12:08.200 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:12:08.213 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T02:12:08.339 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T02:12:08.353 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T02:12:08.380 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:12:08.497 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:12:08.501 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:12:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:12:08.657 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T02:12:08.765 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:12:08.844 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:12:08.855 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T02:12:09.050 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:12:09.075 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T02:12:09.095 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:12:09.123 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:12:09.221 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:09.249 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T02:12:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:12:09.309 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T02:12:09.452 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:12:09.522 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:12:09.533 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:12:09.577 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:09.668 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:12:09.675 INFO:teuthology.orchestra.run.smithi118.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T02:12:09.719 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T02:12:09.832 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:12:09.912 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:12:09.969 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:12:10.039 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:12:10.045 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T02:12:10.270 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:12:10.405 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T02:12:10.415 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:12:10.521 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:12:10.631 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:12:10.649 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T02:12:10.665 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T02:12:10.732 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:10.775 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T02:12:10.804 INFO:teuthology.orchestra.run.smithi003.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T02:12:10.816 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:10.971 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:12:11.033 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:12:11.034 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:11.109 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T02:12:11.159 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:12:11.160 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:12:11.222 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:11.356 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T02:12:11.410 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T02:12:11.503 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:12:11.586 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:12:11.599 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:12:11.709 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:12:11.804 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:12:11.826 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:12:11.854 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T02:12:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T02:12:12.072 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T02:12:12.119 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:12:12.198 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T02:12:12.214 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:12.315 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:12:12.378 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T02:12:12.466 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T02:12:12.599 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-16T02:12:12.630 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:12.641 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:12:12.739 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:12:12.802 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:12:12.859 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:12.950 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:12:12.968 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T02:12:13.032 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:12:13.078 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:12:13.220 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T02:12:13.260 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T02:12:13.266 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T02:12:13.362 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T02:12:13.386 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T02:12:13.455 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T02:12:13.501 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T02:12:13.538 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T02:12:13.670 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T02:12:13.731 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:12:13.785 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:12:13.849 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T02:12:14.040 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:12:14.044 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T02:12:14.058 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:12:14.108 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T02:12:14.185 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:14.300 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:12:14.310 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:14.320 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:12:14.367 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:14.437 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:12:14.446 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T02:12:14.493 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T02:12:14.677 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T02:12:14.722 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:12:14.773 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:14.795 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T02:12:14.823 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:12:14.899 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:14.987 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:12:15.000 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T02:12:15.054 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T02:12:15.091 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:12:15.339 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:12:15.347 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T02:12:15.384 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:12:15.457 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:15.549 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:15.583 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:15.614 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T02:12:15.635 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:12:15.709 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:12:15.735 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:15.761 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:12:15.862 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:15.888 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:15.916 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T02:12:15.975 INFO:teuthology.orchestra.run.smithi177.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T02:12:16.045 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:16.163 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.175 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T02:12:16.230 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:12:16.272 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T02:12:16.276 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T02:12:16.356 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:12:16.538 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.549 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.561 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T02:12:16.658 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.664 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.687 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T02:12:16.749 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:12:16.784 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.790 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:16.916 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:12:17.010 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:17.010 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:17.013 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T02:12:17.025 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T02:12:17.151 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T02:12:17.322 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T02:12:17.410 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:17.477 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:17.553 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:12:17.603 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:17.615 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-16T02:12:17.729 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:17.741 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-16T02:12:17.851 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T02:12:17.855 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:12:17.879 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:12:17.924 INFO:teuthology.orchestra.run.smithi195.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T02:12:18.077 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:18.172 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:12:18.194 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T02:12:18.197 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:18.298 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:18.320 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T02:12:18.423 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T02:12:18.424 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:18.524 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T02:12:18.550 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:18.721 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T02:12:18.775 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T02:12:18.997 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-16T02:12:19.059 INFO:teuthology.orchestra.run.smithi118.stdout:Adding group ceph....done 2024-09-16T02:12:19.092 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:19.120 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:12:19.157 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T02:12:19.227 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:19.353 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:19.388 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:12:19.436 INFO:teuthology.orchestra.run.smithi118.stdout:Adding system user ceph....done 2024-09-16T02:12:19.479 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:19.529 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T02:12:19.587 INFO:teuthology.orchestra.run.smithi118.stdout:Setting system user ceph properties....done 2024-09-16T02:12:19.596 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:19.603 INFO:teuthology.orchestra.run.smithi118.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T02:12:19.634 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T02:12:19.739 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T02:12:19.781 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T02:12:19.960 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T02:12:20.057 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:12:20.081 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T02:12:20.106 INFO:teuthology.orchestra.run.smithi003.stdout:Adding group ceph....done 2024-09-16T02:12:20.261 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T02:12:20.401 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:12:20.499 INFO:teuthology.orchestra.run.smithi003.stdout:Adding system user ceph....done 2024-09-16T02:12:20.518 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:20.527 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:20.546 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T02:12:20.653 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:20.668 INFO:teuthology.orchestra.run.smithi003.stdout:Setting system user ceph properties....done 2024-09-16T02:12:20.669 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:20.687 INFO:teuthology.orchestra.run.smithi003.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T02:12:20.697 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T02:12:20.804 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:12:20.828 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T02:12:20.973 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T02:12:21.140 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.158 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:21.158 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:21.173 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T02:12:21.258 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.343 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T02:12:21.367 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T02:12:21.611 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T02:12:21.630 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.640 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.644 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.733 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.770 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.896 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:21.996 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T02:12:22.086 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T02:12:22.108 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:22.108 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:22.181 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:22.181 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:22.289 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T02:12:22.515 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.572 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.617 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.641 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T02:12:22.673 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.742 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:22.742 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:22.743 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.799 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.844 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:22.934 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:12:22.961 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-16T02:12:22.983 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T02:12:23.183 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.297 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.381 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:23.381 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:23.424 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.435 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.549 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T02:12:23.679 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:23.680 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:23.818 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.859 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:23.944 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.070 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.095 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.132 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:24.132 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:24.154 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.221 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.330 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:24.330 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T02:12:24.347 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.375 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:24.376 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:24.473 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.599 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.636 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.796 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.820 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.821 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.947 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:24.964 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:25.073 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:25.081 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:25.081 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:25.125 INFO:teuthology.orchestra.run.smithi177.stdout:Adding group ceph....done 2024-09-16T02:12:25.199 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T02:12:25.214 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:25.215 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:25.451 INFO:teuthology.orchestra.run.smithi177.stdout:Adding system user ceph....done 2024-09-16T02:12:25.565 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:25.620 INFO:teuthology.orchestra.run.smithi177.stdout:Setting system user ceph properties....done 2024-09-16T02:12:25.635 INFO:teuthology.orchestra.run.smithi177.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T02:12:25.691 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:25.787 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T02:12:25.842 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:25.857 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.080 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:26.080 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:26.124 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T02:12:26.144 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.387 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.428 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.554 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.563 INFO:teuthology.orchestra.run.smithi118.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.577 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.625 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.680 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.728 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.751 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.805 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.854 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.932 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:26.932 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T02:12:27.005 INFO:teuthology.orchestra.run.smithi003.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:27.025 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T02:12:27.177 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T02:12:27.222 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:27.222 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:27.341 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T02:12:27.407 INFO:teuthology.orchestra.run.smithi195.stdout:Adding group ceph....done 2024-09-16T02:12:27.442 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T02:12:27.586 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T02:12:27.678 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:27.801 INFO:teuthology.orchestra.run.smithi195.stdout:Adding system user ceph....done 2024-09-16T02:12:27.868 INFO:teuthology.orchestra.run.smithi118.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T02:12:27.991 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T02:12:27.995 INFO:teuthology.orchestra.run.smithi195.stdout:Setting system user ceph properties....done 2024-09-16T02:12:28.011 INFO:teuthology.orchestra.run.smithi195.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T02:12:28.144 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T02:12:28.438 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:28.470 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T02:12:28.525 INFO:teuthology.orchestra.run.smithi118.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T02:12:28.531 DEBUG:teuthology.parallel:result is None 2024-09-16T02:12:28.660 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:28.660 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:28.812 INFO:teuthology.orchestra.run.smithi003.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T02:12:28.966 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:29.093 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:29.240 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:29.462 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:29.462 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:29.466 INFO:teuthology.orchestra.run.smithi003.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T02:12:29.471 DEBUG:teuthology.parallel:result is None 2024-09-16T02:12:29.550 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:29.550 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T02:12:29.916 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:30.019 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:30.196 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:30.196 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:30.375 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T02:12:30.693 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:30.802 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:30.854 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:30.928 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:31.094 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:31.094 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T02:12:31.124 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:31.124 INFO:teuthology.orchestra.run.smithi177.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:31.581 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:31.604 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:31.747 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:31.802 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:31.802 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T02:12:31.873 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:32.024 INFO:teuthology.orchestra.run.smithi177.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:32.274 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:32.385 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T02:12:32.486 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T02:12:32.558 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:32.559 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T02:12:32.638 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T02:12:33.059 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:33.185 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:33.345 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:33.577 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:33.577 INFO:teuthology.orchestra.run.smithi195.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T02:12:33.772 INFO:teuthology.orchestra.run.smithi177.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T02:12:34.079 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:34.230 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:34.340 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:34.425 INFO:teuthology.orchestra.run.smithi177.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T02:12:34.430 DEBUG:teuthology.parallel:result is None 2024-09-16T02:12:34.491 INFO:teuthology.orchestra.run.smithi195.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-16T02:12:35.182 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-16T02:12:35.307 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T02:12:35.460 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T02:12:36.736 INFO:teuthology.orchestra.run.smithi195.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-16T02:12:37.380 INFO:teuthology.orchestra.run.smithi195.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T02:12:37.385 DEBUG:teuthology.parallel:result is None 2024-09-16T02:12:37.385 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:12:37.534 DEBUG:teuthology.orchestra.run.smithi003:> dpkg-query -W -f '${Version}' ceph 2024-09-16T02:12:37.554 INFO:teuthology.orchestra.run.smithi003.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.555 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.555 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T02:12:37.557 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:12:37.699 DEBUG:teuthology.orchestra.run.smithi118:> dpkg-query -W -f '${Version}' ceph 2024-09-16T02:12:37.718 INFO:teuthology.orchestra.run.smithi118.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.719 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.719 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T02:12:37.721 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:12:37.854 DEBUG:teuthology.orchestra.run.smithi177:> dpkg-query -W -f '${Version}' ceph 2024-09-16T02:12:37.873 INFO:teuthology.orchestra.run.smithi177.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.874 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:37.874 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T02:12:37.876 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-16T02:12:38.020 DEBUG:teuthology.orchestra.run.smithi195:> dpkg-query -W -f '${Version}' ceph 2024-09-16T02:12:38.039 INFO:teuthology.orchestra.run.smithi195.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:38.039 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-16T02:12:38.039 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-16T02:12:38.041 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T02:12:38.041 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:38.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T02:12:38.056 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:38.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T02:12:38.071 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:38.071 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T02:12:38.085 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:38.085 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T02:12:38.100 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T02:12:38.100 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:38.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T02:12:38.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T02:12:38.166 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:38.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T02:12:38.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T02:12:38.233 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:38.233 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T02:12:38.246 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T02:12:38.300 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:38.300 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T02:12:38.313 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T02:12:38.366 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T02:12:38.366 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:38.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T02:12:38.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T02:12:38.434 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:38.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T02:12:38.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T02:12:38.502 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:38.502 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T02:12:38.516 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T02:12:38.576 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:38.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T02:12:38.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T02:12:38.642 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T02:12:38.643 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:38.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T02:12:38.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T02:12:38.711 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:38.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T02:12:38.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T02:12:38.782 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:38.783 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T02:12:38.796 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T02:12:38.855 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:38.855 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T02:12:38.869 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T02:12:38.922 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T02:12:39.005 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T02:12:39.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-09-16T02:12:39.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /var/log/ceph 2024-09-16T02:12:39.009 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /var/log/ceph 2024-09-16T02:12:39.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /var/log/ceph 2024-09-16T02:12:39.025 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T02:12:39.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T02:12:39.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T02:12:39.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T02:12:39.070 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T02:12:39.084 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T02:12:39.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T02:12:39.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T02:12:39.127 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T02:12:39.129 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T02:12:39.153 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-16T02:12:39.153 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T02:12:39.153 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743', 'branch': 'main', 'description': 'rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905743', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOto7MfRAfS9JbCWm9JLp5uccsJFf/ne+FZTN65vjqA6GnS05+xcig1k8Rtqxoot0Ngs7Eb7h22jT60ePNPGcEs=', 'smithi118.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPGa2xV9hiF6j9JMIRLjLhG2E2W8r4Lui6MMuyYZ/r0gDu0Sub0NuLmR0nWxrKfnxf45A65YdgCG6REwWz5Vg98=', 'smithi177.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBVExJro8+FA2WStaHpJpkExYYGfjEYa5dqrRETgYswK3+cbtn6KB6kCY9OVdOZhh71ouIVzlVdAtCtLQ8aRGfw=', 'smithi195.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKFQAQe1MhX6xJ+w/15Zm1YkoOIcxz4zHSnH6VP+WKiFgybZyFk1Y4/r0i+ZSMDr3ANOj1H9P11goliuLZPWgxU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'cluster': 'ceph'}}, {'thrashosds': {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T02:12:39.154 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T02:12:39.180 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T02:12:39.182 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T02:12:39.190 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-16T02:12:39.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T02:12:39.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T02:12:39.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T02:12:39.238 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T02:12:39.251 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:39.251 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T02:12:39.287 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T02:12:39.287 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-09-16T02:12:39.335 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T02:12:39.335 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:39.335 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 873 Links: 1 2024-09-16T02:12:39.336 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:39.336 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-16 02:06:51.048220098 +0000 2024-09-16T02:12:39.336 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-16 02:06:51.044220134 +0000 2024-09-16T02:12:39.336 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-16 02:06:51.044220134 +0000 2024-09-16T02:12:39.336 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-09-16T02:12:39.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T02:12:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-16T02:12:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-16T02:12:39.390 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000379512 s, 1.3 MB/s 2024-09-16T02:12:39.391 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T02:12:39.441 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 890 Links: 1 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-16 02:06:51.828213131 +0000 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-16 02:06:51.536215740 +0000 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-16 02:06:51.536215740 +0000 2024-09-16T02:12:39.488 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-09-16T02:12:39.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T02:12:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-16T02:12:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-16T02:12:39.542 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000416934 s, 1.2 MB/s 2024-09-16T02:12:39.543 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T02:12:39.593 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-16 02:06:52.304208879 +0000 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-16 02:06:52.056211094 +0000 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-16 02:06:52.056211094 +0000 2024-09-16T02:12:39.640 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-09-16T02:12:39.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T02:12:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-16T02:12:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-16T02:12:39.694 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.00041693 s, 1.2 MB/s 2024-09-16T02:12:39.696 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T02:12:39.745 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-09-16T02:12:39.791 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T02:12:39.791 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:39.791 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 925 Links: 1 2024-09-16T02:12:39.791 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:39.791 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-09-16 02:06:52.808204378 +0000 2024-09-16T02:12:39.792 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-09-16 02:06:52.544206736 +0000 2024-09-16T02:12:39.792 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-09-16 02:06:52.544206736 +0000 2024-09-16T02:12:39.792 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: - 2024-09-16T02:12:39.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T02:12:39.845 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-09-16T02:12:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-09-16T02:12:39.846 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000381987 s, 1.3 MB/s 2024-09-16T02:12:39.847 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T02:12:39.897 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:39.897 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:39.897 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T02:12:39.902 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T02:12:39.903 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 876 Links: 1 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-16 02:06:51.829134757 +0000 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-16 02:06:51.573137036 +0000 2024-09-16T02:12:39.951 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-16 02:06:51.573137036 +0000 2024-09-16T02:12:39.952 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-16T02:12:39.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T02:12:40.005 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-16T02:12:40.005 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-16T02:12:40.005 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000332108 s, 1.5 MB/s 2024-09-16T02:12:40.006 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T02:12:40.055 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-09-16T02:12:40.102 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T02:12:40.102 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-16 02:06:52.329130306 +0000 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-16 02:06:52.081132513 +0000 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-16 02:06:52.081132513 +0000 2024-09-16T02:12:40.103 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-16T02:12:40.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T02:12:40.157 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-16T02:12:40.157 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-16T02:12:40.157 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000262412 s, 2.0 MB/s 2024-09-16T02:12:40.158 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T02:12:40.208 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-09-16T02:12:40.254 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T02:12:40.254 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.254 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 905 Links: 1 2024-09-16T02:12:40.254 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.255 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-16 02:06:52.849125679 +0000 2024-09-16T02:12:40.255 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-16 02:06:52.565128207 +0000 2024-09-16T02:12:40.255 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-16 02:06:52.565128207 +0000 2024-09-16T02:12:40.255 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-16T02:12:40.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T02:12:40.309 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-16T02:12:40.309 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-16T02:12:40.309 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000250317 s, 2.0 MB/s 2024-09-16T02:12:40.310 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T02:12:40.360 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-09-16T02:12:40.406 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T02:12:40.406 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.406 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 923 Links: 1 2024-09-16T02:12:40.406 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.407 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-09-16 02:06:53.357121157 +0000 2024-09-16T02:12:40.407 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-09-16 02:06:53.093123507 +0000 2024-09-16T02:12:40.407 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-09-16 02:06:53.093123507 +0000 2024-09-16T02:12:40.407 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: - 2024-09-16T02:12:40.407 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T02:12:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-09-16T02:12:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-09-16T02:12:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000313428 s, 1.6 MB/s 2024-09-16T02:12:40.462 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T02:12:40.512 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:40.513 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:40.513 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T02:12:40.519 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T02:12:40.519 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 02:06:48.632296756 +0000 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 02:06:48.364299149 +0000 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 02:06:48.364299149 +0000 2024-09-16T02:12:40.565 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-16T02:12:40.566 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T02:12:40.619 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T02:12:40.620 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T02:12:40.620 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.00045269 s, 1.1 MB/s 2024-09-16T02:12:40.621 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T02:12:40.674 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-09-16T02:12:40.725 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T02:12:40.726 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.726 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-09-16T02:12:40.726 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.726 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 02:06:49.100292576 +0000 2024-09-16T02:12:40.726 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 02:06:48.848294826 +0000 2024-09-16T02:12:40.727 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 02:06:48.848294826 +0000 2024-09-16T02:12:40.727 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-16T02:12:40.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T02:12:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T02:12:40.779 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T02:12:40.780 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000403584 s, 1.3 MB/s 2024-09-16T02:12:40.781 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T02:12:40.830 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 02:06:49.576288323 +0000 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 02:06:49.312290682 +0000 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 02:06:49.312290682 +0000 2024-09-16T02:12:40.877 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-16T02:12:40.877 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T02:12:40.931 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T02:12:40.931 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T02:12:40.931 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000423626 s, 1.2 MB/s 2024-09-16T02:12:40.932 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T02:12:40.987 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-09-16T02:12:41.037 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-09-16 02:06:50.064283965 +0000 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-09-16 02:06:49.804286288 +0000 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-09-16 02:06:49.804286288 +0000 2024-09-16T02:12:41.038 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: - 2024-09-16T02:12:41.038 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T02:12:41.091 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-09-16T02:12:41.091 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-09-16T02:12:41.091 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000313425 s, 1.6 MB/s 2024-09-16T02:12:41.092 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T02:12:41.144 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:41.144 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:41.144 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T02:12:41.150 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T02:12:41.150 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-16 02:06:58.261632258 +0000 2024-09-16T02:12:41.200 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-16 02:06:58.009634510 +0000 2024-09-16T02:12:41.201 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-16 02:06:58.009634510 +0000 2024-09-16T02:12:41.201 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-09-16T02:12:41.201 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T02:12:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-16T02:12:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-16T02:12:41.254 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000315581 s, 1.6 MB/s 2024-09-16T02:12:41.255 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T02:12:41.304 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-09-16T02:12:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T02:12:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 906 Links: 1 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-16 02:06:58.741627971 +0000 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-16 02:06:58.509630043 +0000 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-16 02:06:58.509630043 +0000 2024-09-16T02:12:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-09-16T02:12:41.353 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T02:12:41.406 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-16T02:12:41.406 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-16T02:12:41.407 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000318988 s, 1.6 MB/s 2024-09-16T02:12:41.408 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T02:12:41.457 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-16 02:06:59.241623504 +0000 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-16 02:06:58.985625792 +0000 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-16 02:06:58.985625792 +0000 2024-09-16T02:12:41.503 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-09-16T02:12:41.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T02:12:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-16T02:12:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-16T02:12:41.556 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000201945 s, 2.5 MB/s 2024-09-16T02:12:41.557 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T02:12:41.604 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-09-16T02:12:41.651 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T02:12:41.652 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T02:12:41.652 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-09-16T02:12:41.652 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T02:12:41.652 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-09-16 02:06:59.733619110 +0000 2024-09-16T02:12:41.653 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-09-16 02:06:59.477621396 +0000 2024-09-16T02:12:41.653 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-09-16 02:06:59.477621396 +0000 2024-09-16T02:12:41.653 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: - 2024-09-16T02:12:41.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T02:12:41.705 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-09-16T02:12:41.705 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-09-16T02:12:41.705 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000293754 s, 1.7 MB/s 2024-09-16T02:12:41.706 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T02:12:41.756 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:41.756 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T02:12:41.756 INFO:tasks.ceph:Generating config... 2024-09-16T02:12:41.757 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T02:12:41.758 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.1 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-16T02:12:41.759 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-16T02:12:41.760 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-16T02:12:41.760 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] ms inject delay type = osd 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-16T02:12:41.760 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-16T02:12:41.760 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T02:12:41.760 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T02:12:41.760 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T02:12:41.760 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T02:12:41.761 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T02:12:41.761 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-16T02:12:41.761 INFO:tasks.ceph:Setting up mon.a... 2024-09-16T02:12:41.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-16T02:12:41.796 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2024-09-16T02:12:41.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-16T02:12:41.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:12:41.943 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.3'), ('mon.b', '172.21.15.118'), ('mon.c', '172.21.15.177')] 2024-09-16T02:12:41.944 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.3,172.21.15.118,172.21.15.177', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-16T02:12:41.944 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:41.944 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T02:12:41.991 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.3 --add b 172.21.15.118 --add c 172.21.15.177 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.079 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:fsid babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2024-09-16T02:12:42.079276+0000 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-09-16T02:12:42.079276+0000 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 17 (quincy) 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 1 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:0: [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon.a 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:1: [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon.b 2024-09-16T02:12:42.080 INFO:teuthology.orchestra.run.smithi003.stdout:2: [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon.c 2024-09-16T02:12:42.081 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-16T02:12:42.082 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T02:12:42.087 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID babbcdda-abe3-4b2f-82b4-da47bc408d34... 2024-09-16T02:12:42.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T02:12:42.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T02:12:42.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T02:12:42.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-16T02:12:42.159 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-09-16T02:12:42.159 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-09-16T02:12:42.159 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T02:12:42.159 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-09-16T02:12:42.159 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-09-16T02:12:42.160 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T02:12:42.161 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "172.21.15.3,172.21.15.118,172.21.15.177" 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon client directed command retry = 5 2024-09-16T02:12:42.162 INFO:teuthology.orchestra.run.smithi003.stdout: mon election default strategy = 1 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject delay max = 1 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject delay probability = 0.005 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject delay type = osd 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject internal delays = 0.002 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject socket failures = 2500 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-09-16T02:12:42.163 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 120 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-09-16T02:12:42.164 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug trim objects = true 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-09-16T02:12:42.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression algorithm = snappy 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-09-16T02:12:42.166 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore write v2 random = True 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore zero block detection = True 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-09-16T02:12:42.167 INFO:teuthology.orchestra.run.smithi003.stdout: osd blocked scrub grace period = 3600 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug reject backfill probability = 0.1 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify cached snaps = True 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify missing on start = True 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd max backfills = 3 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd max markdown count = 1000 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock override recovery settings = True 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue = debug_random 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue cut off = debug_random 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub min interval = 60 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: osd shutdown pgref assert = True 2024-09-16T02:12:42.168 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-09-16T02:12:42.169 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T02:12:42.170 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub interval = 300 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T02:12:42.171 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-09-16T02:12:42.172 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-09-16T02:12:42.172 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout:[global] 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: chdir = "" 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: auth supported = cephx 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: filestore xattr use omap = true 2024-09-16T02:12:42.173 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: mon clock drift allowed = 1.000 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: osd crush chooseleaf type = 0 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: auth debug = true 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.174 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on old message = true 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: ms die on bug = true 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon pg warn max object skew = 0 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default size = 2 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow primary affinity = true 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd allow pg remap = true 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on legacy crush tunables = false 2024-09-16T02:12:42.175 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on crush straw calc version zero = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on no sortbitwise = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on osd down out interval zero = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon warn on too few osds = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon_allow_pool_size_one = true 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: osd default data pool replay window = 5 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon allow pool delete = true 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.176 INFO:teuthology.orchestra.run.smithi177.stdout: mon cluster log file level = debug 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: debug asserts on shutdown = true 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: mon health detail to clog = false 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: mon host = "172.21.15.3,172.21.15.118,172.21.15.177" 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: mon client directed command retry = 5 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: mon election default strategy = 1 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay max = 1 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay probability = 0.005 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject delay type = osd 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject internal delays = 0.002 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: ms inject socket failures = 2500 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T02:12:42.177 INFO:teuthology.orchestra.run.smithi177.stdout: fsid = babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout:[osd] 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd journal size = 100 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub load threshold = 5.0 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub max interval = 120 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock profile = high_recovery_ops 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd recover clone overlap = true 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd recovery max chunk = 1048576 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.178 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug shutdown = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug op order = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify stray on activate = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug trim objects = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd open classes on start = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug pg log writeout = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd deep scrub update digest min age = 30 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: osd map max advance = 10 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: journal zero on create = true 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.179 INFO:teuthology.orchestra.run.smithi177.stdout: filestore ondisk finisher threads = 3 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: filestore apply finisher threads = 3 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bdev debug aio = true 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug misdirected ops = true 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: debug monc = 20 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore block size = 96636764160 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression algorithm = snappy 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore compression mode = aggressive 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore fsck on mount = True 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore write v2 random = True 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: bluestore zero block detection = True 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluefs = 20 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: debug bluestore = 20 2024-09-16T02:12:42.180 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: debug osd = 20 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: debug rocksdb = 10 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd full ratio = 0.9 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd nearfull ratio = 0.8 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd blocked scrub grace period = 3600 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug reject backfill probability = 0.1 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify cached snaps = True 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd debug verify missing on start = True 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd failsafe full ratio = 0.95 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd max backfills = 3 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd max markdown count = 1000 2024-09-16T02:12:42.181 INFO:teuthology.orchestra.run.smithi177.stdout: osd mclock override recovery settings = True 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: osd objectstore = bluestore 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue = debug_random 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: osd op queue cut off = debug_random 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: osd scrub min interval = 60 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: osd shutdown pgref assert = True 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr] 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: debug mgr = 20 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.182 INFO:teuthology.orchestra.run.smithi177.stdout: mgr/telemetry/nag = false 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout:[mon] 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: debug ms = 1 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: debug mon = 20 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: debug paxos = 20 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: debug auth = 20 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon data avail warn = 5 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon mgr mkfs grace = 240 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd reporter subtree level = osd 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon osd prime pg temp = true 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.183 INFO:teuthology.orchestra.run.smithi177.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: # 1m isn't quite enough 2024-09-16T02:12:42.184 INFO:teuthology.orchestra.run.smithi177.stdout: mon_down_mkfs_grace = 2m 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: mon_warn_on_filestore_osds = false 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: mon scrub interval = 300 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout:[client] 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: rgw cache enabled = true 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable ops log = true 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: rgw enable usage log = true 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.a] 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.b] 2024-09-16T02:12:42.185 INFO:teuthology.orchestra.run.smithi177.stdout:[mon.c] 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout:[global] 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: chdir = "" 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: auth supported = cephx 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: filestore xattr use omap = true 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: mon clock drift allowed = 1.000 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: osd crush chooseleaf type = 0 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: auth debug = true 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on old message = true 2024-09-16T02:12:42.186 INFO:teuthology.orchestra.run.smithi195.stdout: ms die on bug = true 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon pg warn max object skew = 0 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default size = 2 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow primary affinity = true 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd allow pg remap = true 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on legacy crush tunables = false 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on crush straw calc version zero = false 2024-09-16T02:12:42.187 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on no sortbitwise = false 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on osd down out interval zero = false 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: mon warn on too few osds = false 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: mon_allow_pool_size_one = true 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.188 INFO:teuthology.orchestra.run.smithi195.stdout: osd default data pool replay window = 5 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon allow pool delete = true 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon cluster log file level = debug 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: debug asserts on shutdown = true 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon health detail to clog = false 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon host = "172.21.15.3,172.21.15.118,172.21.15.177" 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon client directed command retry = 5 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: mon election default strategy = 1 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay max = 1 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay probability = 0.005 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject delay type = osd 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject internal delays = 0.002 2024-09-16T02:12:42.189 INFO:teuthology.orchestra.run.smithi195.stdout: ms inject socket failures = 2500 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: fsid = babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout:[osd] 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd journal size = 100 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub load threshold = 5.0 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub max interval = 120 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock profile = high_recovery_ops 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd recover clone overlap = true 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd recovery max chunk = 1048576 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug shutdown = true 2024-09-16T02:12:42.190 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug op order = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify stray on activate = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug trim objects = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd open classes on start = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug pg log writeout = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd deep scrub update digest min age = 30 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: osd map max advance = 10 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: journal zero on create = true 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: filestore ondisk finisher threads = 3 2024-09-16T02:12:42.191 INFO:teuthology.orchestra.run.smithi195.stdout: filestore apply finisher threads = 3 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bdev debug aio = true 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug misdirected ops = true 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: debug monc = 20 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore block size = 96636764160 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression algorithm = snappy 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore compression mode = aggressive 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore fsck on mount = True 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore write v2 random = True 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: bluestore zero block detection = True 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluefs = 20 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: debug bluestore = 20 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-09-16T02:12:42.192 INFO:teuthology.orchestra.run.smithi195.stdout: debug osd = 20 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: debug rocksdb = 10 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd full ratio = 0.9 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd nearfull ratio = 0.8 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd blocked scrub grace period = 3600 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug reject backfill probability = 0.1 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify cached snaps = True 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd debug verify missing on start = True 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd failsafe full ratio = 0.95 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd max backfills = 3 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd max markdown count = 1000 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd mclock override recovery settings = True 2024-09-16T02:12:42.193 INFO:teuthology.orchestra.run.smithi195.stdout: osd objectstore = bluestore 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue = debug_random 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: osd op queue cut off = debug_random 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: osd scrub min interval = 60 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: osd shutdown pgref assert = True 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout:[mgr] 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: debug mgr = 20 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-09-16T02:12:42.194 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mgr/telemetry/nag = false 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout:[mon] 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: debug ms = 1 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: debug mon = 20 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: debug paxos = 20 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: debug auth = 20 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon data avail warn = 5 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon mgr mkfs grace = 240 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd reporter subtree level = osd 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon osd prime pg temp = true 2024-09-16T02:12:42.195 INFO:teuthology.orchestra.run.smithi195.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.196 INFO:teuthology.orchestra.run.smithi195.stdout: # 1m isn't quite enough 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: mon_down_mkfs_grace = 2m 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: mon_warn_on_filestore_osds = false 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: mon scrub interval = 300 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout:[client] 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: rgw cache enabled = true 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable ops log = true 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: rgw enable usage log = true 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.a] 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.b] 2024-09-16T02:12:42.197 INFO:teuthology.orchestra.run.smithi195.stdout:[mon.c] 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: chdir = "" 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: auth supported = cephx 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: filestore xattr use omap = true 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.199 INFO:teuthology.orchestra.run.smithi118.stdout: mon clock drift allowed = 1.000 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: osd crush chooseleaf type = 0 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: auth debug = true 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on old message = true 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: ms die on bug = true 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: mon pg warn max object skew = 0 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T02:12:42.200 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default size = 2 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow primary affinity = true 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd allow pg remap = true 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on legacy crush tunables = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on crush straw calc version zero = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on no sortbitwise = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on osd down out interval zero = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon warn on too few osds = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T02:12:42.201 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon_allow_pool_size_one = true 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: osd default data pool replay window = 5 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon allow pool delete = true 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon cluster log file level = debug 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: debug asserts on shutdown = true 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon health detail to clog = false 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon host = "172.21.15.3,172.21.15.118,172.21.15.177" 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon client directed command retry = 5 2024-09-16T02:12:42.202 INFO:teuthology.orchestra.run.smithi118.stdout: mon election default strategy = 1 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay max = 1 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay probability = 0.005 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject delay type = osd 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject internal delays = 0.002 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: ms inject socket failures = 2500 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout:[osd] 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd journal size = 100 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub load threshold = 5.0 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub max interval = 120 2024-09-16T02:12:42.203 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock profile = high_recovery_ops 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd recover clone overlap = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd recovery max chunk = 1048576 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug shutdown = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug op order = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify stray on activate = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug trim objects = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd open classes on start = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug pg log writeout = true 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: osd deep scrub update digest min age = 30 2024-09-16T02:12:42.204 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: osd map max advance = 10 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: journal zero on create = true 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: filestore ondisk finisher threads = 3 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: filestore apply finisher threads = 3 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: bdev debug aio = true 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug misdirected ops = true 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: debug monc = 20 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore block size = 96636764160 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression algorithm = snappy 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore compression mode = aggressive 2024-09-16T02:12:42.205 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore fsck on mount = True 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore write v2 random = True 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: bluestore zero block detection = True 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluefs = 20 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: debug bluestore = 20 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: debug osd = 20 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: debug rocksdb = 10 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd full ratio = 0.9 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd nearfull ratio = 0.8 2024-09-16T02:12:42.206 INFO:teuthology.orchestra.run.smithi118.stdout: osd blocked scrub grace period = 3600 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug reject backfill probability = 0.1 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify cached snaps = True 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd debug verify missing on start = True 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd failsafe full ratio = 0.95 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd max backfills = 3 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd max markdown count = 1000 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd mclock override recovery settings = True 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd objectstore = bluestore 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue = debug_random 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd op queue cut off = debug_random 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd scrub min interval = 60 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: osd shutdown pgref assert = True 2024-09-16T02:12:42.207 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr] 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: debug mgr = 20 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: mgr/telemetry/nag = false 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout:[mon] 2024-09-16T02:12:42.208 INFO:teuthology.orchestra.run.smithi118.stdout: debug ms = 1 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: debug mon = 20 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: debug paxos = 20 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: debug auth = 20 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon data avail warn = 5 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon mgr mkfs grace = 240 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min pgs per osd = 4 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd reporter subtree level = osd 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon osd prime pg temp = true 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: mon reweight min bytes per osd = 10 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T02:12:42.209 INFO:teuthology.orchestra.run.smithi118.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: # 1m isn't quite enough 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: mon_down_mkfs_grace = 2m 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: mon_warn_on_filestore_osds = false 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: mon scrub interval = 300 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout:[client] 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: rgw cache enabled = true 2024-09-16T02:12:42.210 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable ops log = true 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout: rgw enable usage log = true 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.a] 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.b] 2024-09-16T02:12:42.211 INFO:teuthology.orchestra.run.smithi118.stdout:[mon.c] 2024-09-16T02:12:42.212 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-16T02:12:42.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-16T02:12:42.296 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T02:12:42.296 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:42.296 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-16T02:12:42.301 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:42.301 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-16T02:12:42.347 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:12:42.347 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:42.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T02:12:42.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:12:42.413 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:12:42.413 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:12:42.459 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:42.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T02:12:42.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:12:42.483 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:12:42.483 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.531 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:12:42.531 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:42.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T02:12:42.531 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:12:42.554 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:12:42.554 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.601 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:12:42.601 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:42.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-16T02:12:42.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-16T02:12:42.623 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:12:42.624 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:42.671 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T02:12:42.671 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T02:12:42.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T02:12:42.718 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-16T02:12:42.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T02:12:42.764 INFO:teuthology.orchestra.run.smithi195.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-16T02:12:42.782 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T02:12:42.782 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T02:12:42.783 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-16T02:12:42.860 INFO:teuthology.orchestra.run.smithi195.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-16T02:12:42.871 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T02:12:42.871 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi118.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi177.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi195.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-16T02:12:42.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-16T02:12:42.886 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:42.886 INFO:tasks.ceph:role: osd.0 2024-09-16T02:12:42.887 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:12:42.887 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:42.951 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:43.411 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-09-16T02:12:43.413 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-09-16T02:12:43.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-16T02:12:43.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-16T02:12:43.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:43.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-16T02:12:43.598 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:43.599 INFO:tasks.ceph:role: osd.4 2024-09-16T02:12:43.599 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:12:43.599 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T02:12:43.663 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:43.663 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:43.664 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:44.094 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-09-16T02:12:44.096 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-09-16T02:12:44.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-16T02:12:44.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-16T02:12:44.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:44.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-16T02:12:44.266 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:44.266 INFO:tasks.ceph:role: osd.8 2024-09-16T02:12:44.266 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:12:44.266 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T02:12:44.333 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:44.334 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:44.796 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-09-16T02:12:44.798 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-09-16T02:12:44.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-16T02:12:44.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-16T02:12:44.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:44.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:44.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-16T02:12:44.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:44.961 INFO:tasks.ceph:role: osd.12 2024-09-16T02:12:44.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:12:44.961 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T02:12:45.022 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:45.023 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:45.484 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-09-16T02:12:45.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-09-16T02:12:45.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-16T02:12:45.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-16T02:12:45.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:45.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:45.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:45.771+0000 7ff1f25a97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-16T02:12:45.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:45.775+0000 7ff1f25a97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-16T02:12:45.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:45.779+0000 7ff1f25a97c0 -1 bdev(0x564002ba3800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-16T02:12:45.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:45.779+0000 7ff1f25a97c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-16T02:12:48.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-16T02:12:48.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:49.023+0000 7fe8e67ac7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-16T02:12:49.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:49.023+0000 7fe8e67ac7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-16T02:12:49.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:49.023+0000 7fe8e67ac7c0 -1 bdev(0x560351f8b800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-16T02:12:49.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:49.023+0000 7fe8e67ac7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-16T02:12:52.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-16T02:12:52.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:52.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:52.299+0000 7f9cd6c037c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-16T02:12:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:52.303+0000 7f9cd6c037c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-16T02:12:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:52.303+0000 7f9cd6c037c0 -1 bdev(0x561de77eb800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-16T02:12:52.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:52.303+0000 7f9cd6c037c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-16T02:12:55.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-16T02:12:55.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:55.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:55.599+0000 7f0218c3f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-16T02:12:55.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:55.599+0000 7f0218c3f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-16T02:12:55.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:55.599+0000 7f0218c3f7c0 -1 bdev(0x55a2b0ce9800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-16T02:12:55.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:12:55.599+0000 7f0218c3f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-16T02:12:58.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-16T02:12:58.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-16T02:12:58.819 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:58.819 INFO:tasks.ceph:role: osd.1 2024-09-16T02:12:58.819 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:12:58.819 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T02:12:58.875 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:58.875 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:58.875 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:58.875 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:58.876 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:58.899 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-16T02:12:58.901 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-16T02:12:58.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-16T02:12:58.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-16T02:12:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:59.018 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:59.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-16T02:12:59.075 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:59.076 INFO:tasks.ceph:role: osd.5 2024-09-16T02:12:59.076 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:12:59.076 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T02:12:59.129 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:59.130 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:59.156 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-16T02:12:59.158 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-16T02:12:59.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-16T02:12:59.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-16T02:12:59.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:59.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-16T02:12:59.334 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:59.334 INFO:tasks.ceph:role: osd.9 2024-09-16T02:12:59.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:12:59.334 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:59.390 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:59.391 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:59.391 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:59.391 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:59.391 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:59.414 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-16T02:12:59.415 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-16T02:12:59.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-16T02:12:59.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-16T02:12:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:59.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-16T02:12:59.581 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-16T02:12:59.582 INFO:tasks.ceph:role: osd.13 2024-09-16T02:12:59.582 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:12:59.582 DEBUG:teuthology.orchestra.run.smithi118:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T02:12:59.638 INFO:teuthology.orchestra.run.smithi118.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:12:59.638 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:12:59.639 INFO:teuthology.orchestra.run.smithi118.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:12:59.662 INFO:teuthology.orchestra.run.smithi118.stdout:Discarding blocks...Done. 2024-09-16T02:12:59.664 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi118.front.sepia.ceph.com -o noatime 2024-09-16T02:12:59.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-16T02:12:59.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-16T02:12:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:12:59.779 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:12:59.779 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:12:59.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:12:59.889+0000 7f71bef227c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-16T02:12:59.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:12:59.889+0000 7f71bef227c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-16T02:12:59.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:12:59.889+0000 7f71bef227c0 -1 bdev(0x5584faadf800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-16T02:12:59.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:12:59.889+0000 7f71bef227c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-16T02:13:03.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-16T02:13:03.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:03.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:03.137+0000 7f962c7537c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-16T02:13:03.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:03.137+0000 7f962c7537c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-16T02:13:03.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:03.137+0000 7f962c7537c0 -1 bdev(0x56141fbed800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:03.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:03.137+0000 7f962c7537c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-16T02:13:06.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-16T02:13:06.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:06.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:06.417+0000 7fdd30ace7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-16T02:13:06.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:06.421+0000 7fdd30ace7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-16T02:13:06.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:06.421+0000 7fdd30ace7c0 -1 bdev(0x55f13e8b5800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:06.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:06.421+0000 7fdd30ace7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-16T02:13:09.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-16T02:13:09.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:09.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:09.669+0000 7fe5e5b1e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-16T02:13:09.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:09.669+0000 7fe5e5b1e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-16T02:13:09.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:09.669+0000 7fe5e5b1e7c0 -1 bdev(0x55f3f617d800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:09.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-09-16T02:13:09.669+0000 7fe5e5b1e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-16T02:13:12.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-16T02:13:12.855 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-16T02:13:12.869 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:12.869 INFO:tasks.ceph:role: osd.2 2024-09-16T02:13:12.869 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:13:12.870 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T02:13:12.931 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:12.931 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:12.932 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:13.378 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T02:13:13.380 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T02:13:13.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-16T02:13:13.476 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-16T02:13:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:13.486 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:13.486 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-16T02:13:13.537 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:13.537 INFO:tasks.ceph:role: osd.6 2024-09-16T02:13:13.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:13:13.537 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:13.599 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:14.061 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T02:13:14.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T02:13:14.063 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-16T02:13:14.202 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-16T02:13:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:14.215 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:14.216 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-16T02:13:14.272 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:14.272 INFO:tasks.ceph:role: osd.10 2024-09-16T02:13:14.272 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:13:14.272 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T02:13:14.335 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:14.336 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:14.812 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T02:13:14.814 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T02:13:14.814 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-16T02:13:14.959 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-16T02:13:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:14.969 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:14.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-16T02:13:15.024 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:15.025 INFO:tasks.ceph:role: osd.14 2024-09-16T02:13:15.025 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:13:15.025 DEBUG:teuthology.orchestra.run.smithi177:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:15.088 INFO:teuthology.orchestra.run.smithi177.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:15.507 INFO:teuthology.orchestra.run.smithi177.stdout:Discarding blocks...Done. 2024-09-16T02:13:15.509 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi177.front.sepia.ceph.com -o noatime 2024-09-16T02:13:15.510 DEBUG:teuthology.orchestra.run.smithi177:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-16T02:13:15.598 DEBUG:teuthology.orchestra.run.smithi177:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-16T02:13:15.607 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:15.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:15.719 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:15.715+0000 7f36487297c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-16T02:13:15.721 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:15.719+0000 7f36487297c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-16T02:13:15.721 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:15.719+0000 7f36487297c0 -1 bdev(0x559ebe54d800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:15.721 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:15.719+0000 7f36487297c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-16T02:13:18.878 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-16T02:13:18.891 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:18.974 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:18.971+0000 7fe2228aa7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-16T02:13:18.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:18.971+0000 7fe2228aa7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-16T02:13:18.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:18.971+0000 7fe2228aa7c0 -1 bdev(0x55e7faacf800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:18.975 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:18.971+0000 7fe2228aa7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-16T02:13:22.111 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-16T02:13:22.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:22.250 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:22.247+0000 7ff20e97d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-16T02:13:22.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:22.247+0000 7ff20e97d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-16T02:13:22.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:22.247+0000 7ff20e97d7c0 -1 bdev(0x55714ff57800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:22.251 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:22.251+0000 7ff20e97d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-16T02:13:25.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-16T02:13:25.428 DEBUG:teuthology.orchestra.run.smithi177:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:25.509 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:25.507+0000 7f3254cf67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-16T02:13:25.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:25.507+0000 7f3254cf67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-16T02:13:25.510 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:25.507+0000 7f3254cf67c0 -1 bdev(0x55adc0979800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:25.511 INFO:teuthology.orchestra.run.smithi177.stderr:2024-09-16T02:13:25.507+0000 7f3254cf67c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-16T02:13:28.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-16T02:13:28.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-16T02:13:28.746 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:28.746 INFO:tasks.ceph:role: osd.3 2024-09-16T02:13:28.746 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:13:28.746 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:28.801 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:28.802 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:28.802 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:28.825 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-09-16T02:13:28.893 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-09-16T02:13:28.893 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-16T02:13:28.951 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-16T02:13:29.004 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:29.005 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-16T02:13:29.058 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:29.058 INFO:tasks.ceph:role: osd.7 2024-09-16T02:13:29.058 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:13:29.058 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T02:13:29.114 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:29.115 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:29.138 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-09-16T02:13:29.140 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-09-16T02:13:29.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-16T02:13:29.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-16T02:13:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:29.255 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:29.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-16T02:13:29.309 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:29.309 INFO:tasks.ceph:role: osd.11 2024-09-16T02:13:29.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:13:29.309 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T02:13:29.361 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:29.362 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:29.385 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-09-16T02:13:29.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-09-16T02:13:29.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-16T02:13:29.450 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-16T02:13:29.503 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:29.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-16T02:13:29.561 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-16T02:13:29.561 INFO:tasks.ceph:role: osd.15 2024-09-16T02:13:29.562 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:13:29.562 DEBUG:teuthology.orchestra.run.smithi195:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout: = sunit=0 swidth=0 blks 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T02:13:29.619 INFO:teuthology.orchestra.run.smithi195.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T02:13:29.642 INFO:teuthology.orchestra.run.smithi195.stdout:Discarding blocks...Done. 2024-09-16T02:13:29.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi195.front.sepia.ceph.com -o noatime 2024-09-16T02:13:29.644 DEBUG:teuthology.orchestra.run.smithi195:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-16T02:13:29.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-16T02:13:29.755 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T02:13:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:13:29.756 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:29.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:29.874+0000 7f44df57a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-16T02:13:29.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:29.874+0000 7f44df57a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-16T02:13:29.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:29.874+0000 7f44df57a7c0 -1 bdev(0x55d01bd2d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:29.875 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:29.874+0000 7f44df57a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-16T02:13:33.012 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-16T02:13:33.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:33.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:33.106+0000 7f7f64b617c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-16T02:13:33.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:33.106+0000 7f7f64b617c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-16T02:13:33.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:33.106+0000 7f7f64b617c0 -1 bdev(0x5646f8ca5800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:33.109 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:33.106+0000 7f7f64b617c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-16T02:13:36.254 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-16T02:13:36.314 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:36.398+0000 7f7594edc7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-16T02:13:36.399 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:36.398+0000 7f7594edc7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-16T02:13:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:36.398+0000 7f7594edc7c0 -1 bdev(0x55fb0a3d1800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:36.400 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:36.398+0000 7f7594edc7c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-16T02:13:39.534 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-16T02:13:39.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:39.670 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:39.670+0000 7ff4aa2c27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-16T02:13:39.671 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:39.670+0000 7ff4aa2c27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-16T02:13:39.671 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:39.670+0000 7ff4aa2c27c0 -1 bdev(0x5595b5b0d800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-16T02:13:39.671 INFO:teuthology.orchestra.run.smithi195.stderr:2024-09-16T02:13:39.670+0000 7ff4aa2c27c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-16T02:13:42.821 DEBUG:teuthology.orchestra.run.smithi195:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-16T02:13:42.874 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T02:13:42.874 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:42.875 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-16T02:13:42.889 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:42.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-16T02:13:42.943 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:42.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-16T02:13:42.998 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:42.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-16T02:13:43.055 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:43.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-16T02:13:43.110 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:43.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-16T02:13:43.125 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:43.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-16T02:13:43.181 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:43.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-16T02:13:43.237 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:43.237 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-16T02:13:43.293 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:43.293 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-16T02:13:43.307 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:43.307 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-16T02:13:43.363 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:43.363 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-16T02:13:43.422 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:43.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-16T02:13:43.474 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-16T02:13:43.487 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.487 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-16T02:13:43.542 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.542 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-16T02:13:43.598 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.598 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-16T02:13:43.653 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.653 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-16T02:13:43.706 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:43.706 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-16T02:13:43.755 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T02:13:43.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T02:13:43.757 DEBUG:teuthology.orchestra.run.smithi118:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T02:13:43.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAalOdmRvnLKhAALd7kneY6I7TwzKdmmcQsFg== 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAdlOdmVixFLhAA+Y6njOv2hN2RtK1Ul6Yklg== 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAhlOdmyWajARAAUNkpicUdqSgTdIIi8jxoHA== 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.8] 2024-09-16T02:13:43.767 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAklOdmgIhAEhAAyvCjL/aLROdGuc6XFYy+ug== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.12] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAnlOdmVZf5IxAAgxMHLth7ME4CJp2adK2GkQ== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQArlOdmPD00NRAAUBvLQq01zHc8kQm0iYhYjQ== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAvlOdmTttpCBAAkL3Jp5LubCNk+RNyxKZ91w== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.9] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAylOdm4+o3GRAAjAgSXYoqljWXoKkOSvbxuw== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.13] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQA1lOdmqHQuKBAARj4bx850ip5cAnmGSyELIQ== 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-09-16T02:13:43.768 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQA7lOdmy2sBKxAA/HTV3rW0zTqsqGgIPwUlcg== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQA+lOdm1VA3OhAAhOOqtyF8WuLCvIPhqxmLZg== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.10] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBClOdmovsWDxAAb9hTI4PVfj8FIFXk+gsssw== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.14] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBFlOdmWviIHhAAa339IyNBircDC43iMriddA== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAalOdm6ISYLRAAnwR+J4ht/cVfouI1mCnqOg== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBJlOdmwlYzNBAAZ9MAM0Hdmq9oTTuYyUve3A== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBNlOdmWrePBhAA8qYoUcp35rcve8ZE3mZYeQ== 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.11] 2024-09-16T02:13:43.769 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBQlOdmrX7hFxAAfyAVDn2/mdXJ+kTyoV0Yuw== 2024-09-16T02:13:43.770 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.15] 2024-09-16T02:13:43.770 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQBTlOdmS2MNKBAAf2MGS8out10rK0Hypn8EFQ== 2024-09-16T02:13:43.770 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-09-16T02:13:43.770 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAalOdmUsBSMxAAlV7Mo4YcFqr80MWgrinbuQ== 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.y] 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAalOdmRvnLKhAALd7kneY6I7TwzKdmmcQsFg== 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.0] 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAdlOdmVixFLhAA+Y6njOv2hN2RtK1Ul6Yklg== 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.4] 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAhlOdmyWajARAAUNkpicUdqSgTdIIi8jxoHA== 2024-09-16T02:13:43.771 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.8] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAklOdmgIhAEhAAyvCjL/aLROdGuc6XFYy+ug== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.12] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAnlOdmVZf5IxAAgxMHLth7ME4CJp2adK2GkQ== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.1] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQArlOdmPD00NRAAUBvLQq01zHc8kQm0iYhYjQ== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.5] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAvlOdmTttpCBAAkL3Jp5LubCNk+RNyxKZ91w== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.9] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAylOdm4+o3GRAAjAgSXYoqljWXoKkOSvbxuw== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.13] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA1lOdmqHQuKBAARj4bx850ip5cAnmGSyELIQ== 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.2] 2024-09-16T02:13:43.772 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA7lOdmy2sBKxAA/HTV3rW0zTqsqGgIPwUlcg== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.6] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQA+lOdm1VA3OhAAhOOqtyF8WuLCvIPhqxmLZg== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.10] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBClOdmovsWDxAAb9hTI4PVfj8FIFXk+gsssw== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.14] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBFlOdmWviIHhAAa339IyNBircDC43iMriddA== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[mgr.x] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAalOdm6ISYLRAAnwR+J4ht/cVfouI1mCnqOg== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.3] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBJlOdmwlYzNBAAZ9MAM0Hdmq9oTTuYyUve3A== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.7] 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBNlOdmWrePBhAA8qYoUcp35rcve8ZE3mZYeQ== 2024-09-16T02:13:43.773 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.11] 2024-09-16T02:13:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBQlOdmrX7hFxAAfyAVDn2/mdXJ+kTyoV0Yuw== 2024-09-16T02:13:43.774 INFO:teuthology.orchestra.run.smithi118.stdout:[osd.15] 2024-09-16T02:13:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQBTlOdmS2MNKBAAf2MGS8out10rK0Hypn8EFQ== 2024-09-16T02:13:43.774 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-09-16T02:13:43.774 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAalOdmUsBSMxAAlV7Mo4YcFqr80MWgrinbuQ== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.y] 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAalOdmRvnLKhAALd7kneY6I7TwzKdmmcQsFg== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.0] 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAdlOdmVixFLhAA+Y6njOv2hN2RtK1Ul6Yklg== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.4] 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAhlOdmyWajARAAUNkpicUdqSgTdIIi8jxoHA== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.8] 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAklOdmgIhAEhAAyvCjL/aLROdGuc6XFYy+ug== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.12] 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAnlOdmVZf5IxAAgxMHLth7ME4CJp2adK2GkQ== 2024-09-16T02:13:43.775 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.1] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQArlOdmPD00NRAAUBvLQq01zHc8kQm0iYhYjQ== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.5] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAvlOdmTttpCBAAkL3Jp5LubCNk+RNyxKZ91w== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.9] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAylOdm4+o3GRAAjAgSXYoqljWXoKkOSvbxuw== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.13] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA1lOdmqHQuKBAARj4bx850ip5cAnmGSyELIQ== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.2] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA7lOdmy2sBKxAA/HTV3rW0zTqsqGgIPwUlcg== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.6] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQA+lOdm1VA3OhAAhOOqtyF8WuLCvIPhqxmLZg== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.10] 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBClOdmovsWDxAAb9hTI4PVfj8FIFXk+gsssw== 2024-09-16T02:13:43.776 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.14] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBFlOdmWviIHhAAa339IyNBircDC43iMriddA== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[mgr.x] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAalOdm6ISYLRAAnwR+J4ht/cVfouI1mCnqOg== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.3] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBJlOdmwlYzNBAAZ9MAM0Hdmq9oTTuYyUve3A== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.7] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBNlOdmWrePBhAA8qYoUcp35rcve8ZE3mZYeQ== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.11] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBQlOdmrX7hFxAAfyAVDn2/mdXJ+kTyoV0Yuw== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[osd.15] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQBTlOdmS2MNKBAAf2MGS8out10rK0Hypn8EFQ== 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout:[client.0] 2024-09-16T02:13:43.777 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQAalOdmUsBSMxAAlV7Mo4YcFqr80MWgrinbuQ== 2024-09-16T02:13:43.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:43.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:43.825 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:43.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:43.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:43.910 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:43.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:43.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:43.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.166 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.215 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.298 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.390 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.474 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.562 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.646 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.818 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:44.897 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:44.906 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T02:13:44.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.980 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:44.994 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.078 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.162 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.233 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.242 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T02:13:45.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T02:13:45.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T02:13:45.330 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T02:13:45.401 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T02:13:45.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-16T02:13:45.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:13:45.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-16T02:13:45.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-16T02:13:45.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:13:45.623 DEBUG:teuthology.orchestra.run.smithi118:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-16T02:13:45.634 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-16T02:13:45.647 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-16T02:13:45.743 DEBUG:teuthology.orchestra.run.smithi177:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-16T02:13:45.754 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:45.757 DEBUG:teuthology.orchestra.run.smithi118:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:45.759 DEBUG:teuthology.orchestra.run.smithi177:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-16T02:13:45.798 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-16T02:13:45.799 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-16T02:13:45.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-16T02:13:45.804 INFO:tasks.ceph.mon.a:Started 2024-09-16T02:13:45.804 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-16T02:13:45.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-16T02:13:45.806 INFO:tasks.ceph.mon.b:Started 2024-09-16T02:13:45.806 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-16T02:13:45.806 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-16T02:13:45.840 INFO:tasks.ceph.mon.c:Started 2024-09-16T02:13:45.841 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-16T02:13:45.841 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-16T02:13:45.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-16T02:13:45.843 INFO:tasks.ceph.mgr.y:Started 2024-09-16T02:13:45.844 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-16T02:13:45.844 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-16T02:13:45.846 INFO:tasks.ceph.mgr.x:Started 2024-09-16T02:13:45.846 DEBUG:tasks.ceph:set 0 configs 2024-09-16T02:13:45.846 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T02:13:45.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2024-09-16T02:13:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-09-16T02:13:51.349 INFO:tasks.ceph:check_enable_crimson: False 2024-09-16T02:13:51.349 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-16T02:13:51.349 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:51.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-16T02:13:51.361 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:51.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-16T02:13:51.412 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:51.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-16T02:13:51.465 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-09-16T02:13:51.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-16T02:13:51.520 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:51.520 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-16T02:13:51.535 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:51.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-16T02:13:51.588 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:51.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-16T02:13:51.644 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-09-16T02:13:51.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-16T02:13:51.701 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:51.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-16T02:13:51.715 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:51.716 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-16T02:13:51.770 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:51.770 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-16T02:13:51.823 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-09-16T02:13:51.823 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-16T02:13:51.878 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:51.878 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-16T02:13:51.891 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:51.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-16T02:13:51.946 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:51.946 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-16T02:13:52.001 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-09-16T02:13:52.001 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-16T02:13:52.053 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new d00dcd0d-0b5f-48c9-8d98-adaaabf5283a 0 2024-09-16T02:13:52.306 INFO:teuthology.orchestra.run.smithi195.stdout:0 2024-09-16T02:13:52.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c 1 2024-09-16T02:13:52.534 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-09-16T02:13:52.551 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 6bfe62d5-9c5b-44df-bc4f-cf312837c932 2 2024-09-16T02:13:52.762 INFO:teuthology.orchestra.run.smithi195.stdout:2 2024-09-16T02:13:52.779 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new fe2b6f49-91ab-4692-ab9c-4b3bda3bf429 3 2024-09-16T02:13:52.982 INFO:teuthology.orchestra.run.smithi195.stdout:3 2024-09-16T02:13:52.998 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 399375fc-ce28-40e2-89b1-e0fc2c1f44ec 4 2024-09-16T02:13:53.209 INFO:teuthology.orchestra.run.smithi195.stdout:4 2024-09-16T02:13:53.228 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3 5 2024-09-16T02:13:53.431 INFO:teuthology.orchestra.run.smithi195.stdout:5 2024-09-16T02:13:53.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new ac6efa66-27ed-494c-956d-22b90be7a8fd 6 2024-09-16T02:13:53.659 INFO:teuthology.orchestra.run.smithi195.stdout:6 2024-09-16T02:13:53.677 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new fb2a81f7-b4f5-4c4c-aff8-801e8b73d192 7 2024-09-16T02:13:53.941 INFO:teuthology.orchestra.run.smithi195.stdout:7 2024-09-16T02:13:53.957 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 01aca5fb-932f-4ecb-a9ab-c716a5247373 8 2024-09-16T02:13:54.047 INFO:tasks.ceph.mgr.x.smithi195.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T02:13:54.048 INFO:tasks.ceph.mgr.x.smithi195.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T02:13:54.048 INFO:tasks.ceph.mgr.x.smithi195.stderr: from numpy import show_config as show_numpy_config 2024-09-16T02:13:54.164 INFO:teuthology.orchestra.run.smithi195.stdout:8 2024-09-16T02:13:54.174 INFO:tasks.ceph.mgr.y.smithi003.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-16T02:13:54.174 INFO:tasks.ceph.mgr.y.smithi003.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-16T02:13:54.174 INFO:tasks.ceph.mgr.y.smithi003.stderr: from numpy import show_config as show_numpy_config 2024-09-16T02:13:54.181 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new d52efa47-5645-4018-b495-ed116da5f7a9 9 2024-09-16T02:13:54.389 INFO:teuthology.orchestra.run.smithi195.stdout:9 2024-09-16T02:13:54.406 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 99005d53-e62d-4c20-8d3c-541fd20cbeb8 10 2024-09-16T02:13:54.622 INFO:teuthology.orchestra.run.smithi195.stdout:10 2024-09-16T02:13:54.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new f460f1c7-0172-4bd9-875f-b34aff85804f 11 2024-09-16T02:13:54.852 INFO:teuthology.orchestra.run.smithi195.stdout:11 2024-09-16T02:13:54.870 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 7bd5fea1-6e94-478a-8a8f-87459b4030a2 12 2024-09-16T02:13:55.074 INFO:teuthology.orchestra.run.smithi195.stdout:12 2024-09-16T02:13:55.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 89c43816-fffc-42a6-98af-7f215631d595 13 2024-09-16T02:13:55.295 INFO:teuthology.orchestra.run.smithi195.stdout:13 2024-09-16T02:13:55.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new 33cb3383-2d68-4928-be43-cfc215f6b080 14 2024-09-16T02:13:55.522 INFO:teuthology.orchestra.run.smithi195.stdout:14 2024-09-16T02:13:55.539 DEBUG:teuthology.orchestra.run.smithi195:> sudo ceph --cluster ceph osd new b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e 15 2024-09-16T02:13:55.741 INFO:teuthology.orchestra.run.smithi195.stdout:15 2024-09-16T02:13:55.758 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-16T02:13:55.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-16T02:13:55.760 INFO:tasks.ceph.osd.0:Started 2024-09-16T02:13:55.760 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-16T02:13:55.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-16T02:13:55.763 INFO:tasks.ceph.osd.4:Started 2024-09-16T02:13:55.763 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-16T02:13:55.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-16T02:13:55.765 INFO:tasks.ceph.osd.8:Started 2024-09-16T02:13:55.765 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-16T02:13:55.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-16T02:13:55.767 INFO:tasks.ceph.osd.12:Started 2024-09-16T02:13:55.768 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-16T02:13:55.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-16T02:13:55.770 INFO:tasks.ceph.osd.1:Started 2024-09-16T02:13:55.770 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-16T02:13:55.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-16T02:13:55.773 INFO:tasks.ceph.osd.5:Started 2024-09-16T02:13:55.773 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-16T02:13:55.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-16T02:13:55.776 INFO:tasks.ceph.osd.9:Started 2024-09-16T02:13:55.776 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-16T02:13:55.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-16T02:13:55.778 INFO:tasks.ceph.osd.13:Started 2024-09-16T02:13:55.778 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-16T02:13:55.778 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-16T02:13:55.781 INFO:tasks.ceph.osd.2:Started 2024-09-16T02:13:55.781 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-16T02:13:55.781 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-16T02:13:55.784 INFO:tasks.ceph.osd.6:Started 2024-09-16T02:13:55.784 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T02:13:55.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T02:13:55.786 INFO:tasks.ceph.osd.10:Started 2024-09-16T02:13:55.786 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-16T02:13:55.787 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-16T02:13:55.789 INFO:tasks.ceph.osd.14:Started 2024-09-16T02:13:55.789 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-16T02:13:55.789 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-16T02:13:55.791 INFO:tasks.ceph.osd.3:Started 2024-09-16T02:13:55.791 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-16T02:13:55.791 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-16T02:13:55.793 INFO:tasks.ceph.osd.7:Started 2024-09-16T02:13:55.794 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-16T02:13:55.794 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-16T02:13:55.796 INFO:tasks.ceph.osd.11:Started 2024-09-16T02:13:55.796 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-16T02:13:55.796 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-16T02:13:55.798 INFO:tasks.ceph.osd.15:Started 2024-09-16T02:13:55.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:13:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:13:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":18,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:13:55.736760+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T02:13:55.736760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:13:56.013 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-16T02:13:56.013 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T02:13:56.898 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:13:56.897+0000 7f452bab47c0 -1 Falling back to public interface 2024-09-16T02:13:56.928 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:13:56.926+0000 7f4a660317c0 -1 Falling back to public interface 2024-09-16T02:13:56.944 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:13:56.942+0000 7f349af977c0 -1 Falling back to public interface 2024-09-16T02:13:56.946 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:13:56.945+0000 7f65295e27c0 -1 Falling back to public interface 2024-09-16T02:13:56.959 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:13:56.958+0000 7f1dc6bae7c0 -1 Falling back to public interface 2024-09-16T02:13:57.004 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-09-16T02:13:57.002+0000 7f276ae18640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-16T02:13:57.013 INFO:tasks.ceph.mgr.x.smithi195.stderr:2024-09-16T02:13:57.010+0000 7f276ae18640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-16T02:13:57.169 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:13:57.166+0000 7f0d59bb87c0 -1 Falling back to public interface 2024-09-16T02:13:57.183 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:13:57.182+0000 7f684fb5b7c0 -1 Falling back to public interface 2024-09-16T02:13:57.220 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:13:57.215+0000 7f24132f67c0 -1 Falling back to public interface 2024-09-16T02:13:57.226 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:13:57.225+0000 7f51b45127c0 -1 Falling back to public interface 2024-09-16T02:13:57.392 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:13:57.387+0000 7f73d16707c0 -1 Falling back to public interface 2024-09-16T02:13:57.395 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:13:57.394+0000 7f3ad235c7c0 -1 Falling back to public interface 2024-09-16T02:13:57.405 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:13:57.402+0000 7f3cf6c087c0 -1 Falling back to public interface 2024-09-16T02:13:57.427 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:13:57.425+0000 7f37985c07c0 -1 Falling back to public interface 2024-09-16T02:13:57.460 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:13:57.455+0000 7f96b4be17c0 -1 Falling back to public interface 2024-09-16T02:13:57.487 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:13:57.483+0000 7f58a30057c0 -1 Falling back to public interface 2024-09-16T02:13:57.626 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T02:13:57.652 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:13:57.650+0000 7f8ff13397c0 -1 Falling back to public interface 2024-09-16T02:13:57.938 INFO:teuthology.misc.health.smithi003.stdout: 2024-09-16T02:13:57.938 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":18,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:13:55.736760+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T02:13:55.736760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:13:57.954 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-16T02:14:00.933 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:00.929+0000 7f1dc6bae7c0 -1 osd.15 0 log_to_monitors true 2024-09-16T02:14:00.952 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:00.949+0000 7f4a660317c0 -1 osd.14 0 log_to_monitors true 2024-09-16T02:14:00.996 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:00.992+0000 7f452bab47c0 -1 osd.5 0 log_to_monitors true 2024-09-16T02:14:01.026 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:01.021+0000 7f349af977c0 -1 osd.10 0 log_to_monitors true 2024-09-16T02:14:01.109 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:01.105+0000 7f684fb5b7c0 -1 osd.7 0 log_to_monitors true 2024-09-16T02:14:01.182 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:01.177+0000 7f0d59bb87c0 -1 osd.2 0 log_to_monitors true 2024-09-16T02:14:01.296 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:01.292+0000 7f65295e27c0 -1 osd.13 0 log_to_monitors true 2024-09-16T02:14:01.342 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:01.340+0000 7f51b45127c0 -1 osd.1 0 log_to_monitors true 2024-09-16T02:14:01.363 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:01.359+0000 7f73d16707c0 -1 osd.0 0 log_to_monitors true 2024-09-16T02:14:01.510 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:01.503+0000 7f24132f67c0 -1 osd.12 0 log_to_monitors true 2024-09-16T02:14:01.565 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:01.561+0000 7f8ff13397c0 -1 osd.3 0 log_to_monitors true 2024-09-16T02:14:01.665 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:01.665+0000 7f3cf6c087c0 -1 osd.6 0 log_to_monitors true 2024-09-16T02:14:01.677 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:01.673+0000 7f3ad235c7c0 -1 osd.11 0 log_to_monitors true 2024-09-16T02:14:01.747 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:01.744+0000 7f37985c07c0 -1 osd.9 0 log_to_monitors true 2024-09-16T02:14:01.747 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:01.743+0000 7f96b4be17c0 -1 osd.8 0 log_to_monitors true 2024-09-16T02:14:01.785 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:01.779+0000 7f58a30057c0 -1 osd.4 0 log_to_monitors true 2024-09-16T02:14:02.027 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:02.025+0000 7f4a61fda640 -1 osd.14 0 waiting for initial osdmap 2024-09-16T02:14:02.033 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:02.029+0000 7f1dc2b57640 -1 osd.15 0 waiting for initial osdmap 2024-09-16T02:14:02.042 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:02.037+0000 7f1dbe991640 -1 osd.15 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:02.043 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:02.041+0000 7f4a5de14640 -1 osd.14 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.035 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:03.033+0000 7f684bb04640 -1 osd.7 0 waiting for initial osdmap 2024-09-16T02:14:03.037 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:03.031+0000 7f73cd619640 -1 osd.0 0 waiting for initial osdmap 2024-09-16T02:14:03.042 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:03.040+0000 7f51b04bb640 -1 osd.1 0 waiting for initial osdmap 2024-09-16T02:14:03.042 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:03.040+0000 7f3794569640 -1 osd.9 0 waiting for initial osdmap 2024-09-16T02:14:03.042 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:03.040+0000 7f4527a5d640 -1 osd.5 0 waiting for initial osdmap 2024-09-16T02:14:03.043 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:03.035+0000 7f96b0b8a640 -1 osd.8 0 waiting for initial osdmap 2024-09-16T02:14:03.043 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:03.039+0000 7f240f29f640 -1 osd.12 0 waiting for initial osdmap 2024-09-16T02:14:03.044 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:03.039+0000 7f589efae640 -1 osd.4 0 waiting for initial osdmap 2024-09-16T02:14:03.044 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:03.045+0000 7f3496f40640 -1 osd.10 0 waiting for initial osdmap 2024-09-16T02:14:03.045 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:03.041+0000 7f8fed2e2640 -1 osd.3 0 waiting for initial osdmap 2024-09-16T02:14:03.051 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:03.048+0000 7f51ac2f5640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.057 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:03.051+0000 7f73c9453640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.059 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:03.055+0000 7f96ac9c4640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.059 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:03.055+0000 7f240b0d9640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.066 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:03.064+0000 7f4523897640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.067 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:03.065+0000 7f684793e640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.068 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:03.064+0000 7f37903a3640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.069 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:03.069+0000 7f3492d7a640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.071 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:03.069+0000 7f8fe911c640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.071 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:03.067+0000 7f589ade8640 -1 osd.4 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.688 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:03.685+0000 7f3aceb17640 -1 osd.11 0 waiting for initial osdmap 2024-09-16T02:14:03.719 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:03.717+0000 7f3ac6137640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.814 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:03.812+0000 7f652558b640 -1 osd.13 0 waiting for initial osdmap 2024-09-16T02:14:03.838 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:03.836+0000 7f651d3bd640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.859 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:03.857+0000 7f0d56373640 -1 osd.2 0 waiting for initial osdmap 2024-09-16T02:14:03.884 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:03.885+0000 7f0d4d993640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:03.895 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:03.893+0000 7f3cf33c3640 -1 osd.6 0 waiting for initial osdmap 2024-09-16T02:14:03.912 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:03.913+0000 7f3cea9e3640 -1 osd.6 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:14:05.567 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-16T02:14:05.885 INFO:teuthology.misc.health.smithi003.stdout: 2024-09-16T02:14:05.885 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":23,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:14:05.038727+0000","last_up_change":"2024-09-16T02:14:04.035127+0000","last_in_change":"2024-09-16T02:13:55.736760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6809","nonce":3769110015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6811","nonce":3769110015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6815","nonce":3769110015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6813","nonce":3769110015}]},"public_addr":"172.21.15.177:6809/3769110015","cluster_addr":"172.21.15.177:6811/3769110015","heartbeat_back_addr":"172.21.15.177:6815/3769110015","heartbeat_front_addr":"172.21.15.177:6813/3769110015","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.390713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.329190+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.172914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.553892+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.795237+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.952515+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.696414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.100374+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.720464+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.793071+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.982210+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.697339+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.478801+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.343858+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.923086+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.888880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:14:05.903 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-16T02:14:05.903 INFO:tasks.ceph:Creating RBD pool 2024-09-16T02:14:05.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-16T02:14:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'rbd' created 2024-09-16T02:14:07.123 DEBUG:teuthology.orchestra.run.smithi003:> rbd --cluster ceph pool init rbd 2024-09-16T02:14:10.204 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-16T02:14:10.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-16T02:14:10.205 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T02:14:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:10.556 INFO:teuthology.orchestra.run.smithi003.stdout:[{"version":5,"timestamp":"2024-09-16T02:14:03.896027+0000","name":"","changes":[{"name":"osd.6/osd_mclock_max_capacity_iops_ssd","new_value":"6056.086385"}]}] 2024-09-16T02:14:10.556 INFO:tasks.ceph_manager:config epoch is 5 2024-09-16T02:14:10.556 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-16T02:14:10.557 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-16T02:14:10.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-16T02:14:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:10.945 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":4166369282},{"type":"v1","addr":"172.21.15.195:6801","nonce":4166369282}]},"active_addr":"172.21.15.195:6801/4166369282","active_change":"2024-09-16T02:13:55.974938+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":2945467751}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":2261801196}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":2528094517}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.195:0","nonce":3752730977}]}]} 2024-09-16T02:14:10.950 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-16T02:14:10.951 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T02:14:10.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:14:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:11.274 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":28,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:14:10.159151+0000","last_up_change":"2024-09-16T02:14:04.035127+0000","last_in_change":"2024-09-16T02:13:55.736760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6809","nonce":3769110015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6811","nonce":3769110015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6815","nonce":3769110015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6813","nonce":3769110015}]},"public_addr":"172.21.15.177:6809/3769110015","cluster_addr":"172.21.15.177:6811/3769110015","heartbeat_back_addr":"172.21.15.177:6815/3769110015","heartbeat_front_addr":"172.21.15.177:6813/3769110015","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.390713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.329190+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.172914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.553892+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.795237+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.952515+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.696414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.100374+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.720464+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.793071+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.982210+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.697339+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.478801+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.343858+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.923086+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.888880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:14:11.294 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T02:14:11.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:14:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:11.619 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":28,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:14:10.159151+0000","last_up_change":"2024-09-16T02:14:04.035127+0000","last_in_change":"2024-09-16T02:13:55.736760+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6809","nonce":3769110015}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6811","nonce":3769110015}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6815","nonce":3769110015}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3769110015},{"type":"v1","addr":"172.21.15.177:6813","nonce":3769110015}]},"public_addr":"172.21.15.177:6809/3769110015","cluster_addr":"172.21.15.177:6811/3769110015","heartbeat_back_addr":"172.21.15.177:6815/3769110015","heartbeat_front_addr":"172.21.15.177:6813/3769110015","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.390713+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.329190+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.172914+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.553892+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.795237+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.952515+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.696414+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.100374+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.720464+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.793071+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.982210+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.697339+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.478801+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:02.343858+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.923086+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:14:01.888880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:14:11.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:14:11.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:14:11.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:14:11.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:14:11.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T02:14:11.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T02:14:11.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T02:14:11.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T02:14:11.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T02:14:11.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T02:14:11.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T02:14:11.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T02:14:11.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T02:14:11.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T02:14:11.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T02:14:11.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T02:14:12.041 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T02:14:12.072 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T02:14:12.094 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T02:14:12.123 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:14:12.125 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T02:14:12.132 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:14:12.144 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T02:14:12.170 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T02:14:12.227 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T02:14:12.230 INFO:teuthology.orchestra.run.smithi003.stdout:90194313219 2024-09-16T02:14:12.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T02:14:12.248 INFO:teuthology.orchestra.run.smithi003.stdout:94489280517 2024-09-16T02:14:12.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:14:12.253 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T02:14:12.257 INFO:teuthology.orchestra.run.smithi003.stdout:90194313221 2024-09-16T02:14:12.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T02:14:12.271 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:14:12.274 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:12.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T02:14:12.287 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:12.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T02:14:12.958 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.011 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.10 2024-09-16T02:14:13.034 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.035 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.088 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.6 2024-09-16T02:14:13.089 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.2 2024-09-16T02:14:13.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:13.130+0000 7fd14c9fd640 0 --2- 172.21.15.3:0/2679507777 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1480cc3e0 0x7fd1480c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-16T02:14:13.149 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.178 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.202 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.207 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.4 2024-09-16T02:14:13.230 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.232 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.0 2024-09-16T02:14:13.241 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.13 2024-09-16T02:14:13.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.11 2024-09-16T02:14:13.270 INFO:teuthology.orchestra.run.smithi003.stdout:94489280514 2024-09-16T02:14:13.286 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.5 2024-09-16T02:14:13.296 INFO:teuthology.orchestra.run.smithi003.stdout:90194313218 2024-09-16T02:14:13.305 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.306 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:13.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.7 2024-09-16T02:14:13.308 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.313 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.320 INFO:teuthology.orchestra.run.smithi003.stdout:90194313220 2024-09-16T02:14:13.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313218 for osd.15 2024-09-16T02:14:13.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.8 2024-09-16T02:14:13.340 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280516 for osd.1 2024-09-16T02:14:13.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.12 2024-09-16T02:14:13.348 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.9 2024-09-16T02:14:13.355 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313220 for osd.14 2024-09-16T02:14:13.447 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:13.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.3 2024-09-16T02:14:14.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T02:14:14.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T02:14:14.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:14:14.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T02:14:14.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:14:14.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T02:14:14.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T02:14:14.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T02:14:14.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T02:14:14.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T02:14:14.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T02:14:14.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:14:14.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T02:14:14.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T02:14:14.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T02:14:14.422 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:14.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:14:14.484 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.10 2024-09-16T02:14:14.484 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:14.630 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:14.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.2 2024-09-16T02:14:14.692 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:14.798 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:14.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.6 2024-09-16T02:14:14.874 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:14.974 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.022 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.4 2024-09-16T02:14:15.034 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.051 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.13 2024-09-16T02:14:15.080 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.090 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.11 2024-09-16T02:14:15.090 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.158 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.167 INFO:teuthology.orchestra.run.smithi003.stdout:94489280517 2024-09-16T02:14:15.192 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.9 2024-09-16T02:14:15.192 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.212 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280517 got 94489280517 for osd.1 2024-09-16T02:14:15.212 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.222 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.242 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.244 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-09-16T02:14:15.245 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.265 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2024-09-16T02:14:15.265 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.269 INFO:teuthology.orchestra.run.smithi003.stdout:94489280515 2024-09-16T02:14:15.275 INFO:teuthology.orchestra.run.smithi003.stdout:90194313219 2024-09-16T02:14:15.284 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.292 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.7 2024-09-16T02:14:15.292 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.303 INFO:teuthology.orchestra.run.smithi003.stdout:90194313221 2024-09-16T02:14:15.304 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.3 2024-09-16T02:14:15.304 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.306 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313219 got 90194313219 for osd.15 2024-09-16T02:14:15.307 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.14 2024-09-16T02:14:15.325 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.327 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.345 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-16T02:14:15.346 INFO:teuthology.orchestra.run.smithi003.stdout:94489280516 2024-09-16T02:14:15.346 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-09-16T02:14:15.363 DEBUG:teuthology.parallel:result is None 2024-09-16T02:14:15.363 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T02:14:15.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:15.700 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:15.700 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:15.717 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-16T02:14:13.993813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":445240,"kb_used_data":4416,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504200,"statfs":{"total":1546188226560,"available":1545732300800,"internally_reserved":0,"allocated":4521984,"data_stored":2980230,"data_compressed":334950,"data_compressed_allocated":475136,"data_compressed_original":1810432,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.823864"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.396734+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:10.396734+0000","last_peered":"2024-09-16T02:14:10.396734+0000","last_clean":"2024-09-16T02:14:10.396734+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:10.396734+0000","last_undegraded":"2024-09-16T02:14:10.396734+0000","last_fullsized":"2024-09-16T02:14:10.396734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.172783+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:10.172783+0000","last_peered":"2024-09-16T02:14:10.172783+0000","last_clean":"2024-09-16T02:14:10.172783+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:10.172783+0000","last_undegraded":"2024-09-16T02:14:10.172783+0000","last_fullsized":"2024-09-16T02:14:10.172783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.393169+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:10.393169+0000","last_peered":"2024-09-16T02:14:10.393169+0000","last_clean":"2024-09-16T02:14:10.393169+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:10.393169+0000","last_undegraded":"2024-09-16T02:14:10.393169+0000","last_fullsized":"2024-09-16T02:14:10.393169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.237483+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:10.237483+0000","last_peered":"2024-09-16T02:14:10.237483+0000","last_clean":"2024-09-16T02:14:10.237483+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:10.237483+0000","last_undegraded":"2024-09-16T02:14:10.237483+0000","last_fullsized":"2024-09-16T02:14:10.237483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.180900+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:10.180900+0000","last_peered":"2024-09-16T02:14:10.180900+0000","last_clean":"2024-09-16T02:14:10.180900+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:10.180900+0000","last_undegraded":"2024-09-16T02:14:10.180900+0000","last_fullsized":"2024-09-16T02:14:10.180900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.239397+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:10.239397+0000","last_peered":"2024-09-16T02:14:10.239397+0000","last_clean":"2024-09-16T02:14:10.239397+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:10.239397+0000","last_undegraded":"2024-09-16T02:14:10.239397+0000","last_fullsized":"2024-09-16T02:14:10.239397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.179194+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:10.179194+0000","last_peered":"2024-09-16T02:14:10.179194+0000","last_clean":"2024-09-16T02:14:10.179194+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:10.179194+0000","last_undegraded":"2024-09-16T02:14:10.179194+0000","last_fullsized":"2024-09-16T02:14:10.179194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.306016+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:10.306016+0000","last_peered":"2024-09-16T02:14:10.306016+0000","last_clean":"2024-09-16T02:14:10.306016+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:10.306016+0000","last_undegraded":"2024-09-16T02:14:10.306016+0000","last_fullsized":"2024-09-16T02:14:10.306016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.238666+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:10.238666+0000","last_peered":"2024-09-16T02:14:10.238666+0000","last_clean":"2024-09-16T02:14:10.238666+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:10.238666+0000","last_undegraded":"2024-09-16T02:14:10.238666+0000","last_fullsized":"2024-09-16T02:14:10.238666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313221,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313219,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:15.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:16.036 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:16.036 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:16.053 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T02:14:15.994531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":445240,"kb_used_data":4416,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504200,"statfs":{"total":1546188226560,"available":1545732300800,"internally_reserved":0,"allocated":4521984,"data_stored":2980230,"data_compressed":334950,"data_compressed_allocated":475136,"data_compressed_original":1810432,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.824582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.396734+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:10.396734+0000","last_peered":"2024-09-16T02:14:10.396734+0000","last_clean":"2024-09-16T02:14:10.396734+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:10.396734+0000","last_undegraded":"2024-09-16T02:14:10.396734+0000","last_fullsized":"2024-09-16T02:14:10.396734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.172783+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:10.172783+0000","last_peered":"2024-09-16T02:14:10.172783+0000","last_clean":"2024-09-16T02:14:10.172783+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:10.172783+0000","last_undegraded":"2024-09-16T02:14:10.172783+0000","last_fullsized":"2024-09-16T02:14:10.172783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.393169+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:10.393169+0000","last_peered":"2024-09-16T02:14:10.393169+0000","last_clean":"2024-09-16T02:14:10.393169+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:10.393169+0000","last_undegraded":"2024-09-16T02:14:10.393169+0000","last_fullsized":"2024-09-16T02:14:10.393169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.237483+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:10.237483+0000","last_peered":"2024-09-16T02:14:10.237483+0000","last_clean":"2024-09-16T02:14:10.237483+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:10.237483+0000","last_undegraded":"2024-09-16T02:14:10.237483+0000","last_fullsized":"2024-09-16T02:14:10.237483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.180900+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:10.180900+0000","last_peered":"2024-09-16T02:14:10.180900+0000","last_clean":"2024-09-16T02:14:10.180900+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:10.180900+0000","last_undegraded":"2024-09-16T02:14:10.180900+0000","last_fullsized":"2024-09-16T02:14:10.180900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.239397+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:10.239397+0000","last_peered":"2024-09-16T02:14:10.239397+0000","last_clean":"2024-09-16T02:14:10.239397+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:10.239397+0000","last_undegraded":"2024-09-16T02:14:10.239397+0000","last_fullsized":"2024-09-16T02:14:10.239397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.179194+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:10.179194+0000","last_peered":"2024-09-16T02:14:10.179194+0000","last_clean":"2024-09-16T02:14:10.179194+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:10.179194+0000","last_undegraded":"2024-09-16T02:14:10.179194+0000","last_fullsized":"2024-09-16T02:14:10.179194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.306016+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:10.306016+0000","last_peered":"2024-09-16T02:14:10.306016+0000","last_clean":"2024-09-16T02:14:10.306016+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:10.306016+0000","last_undegraded":"2024-09-16T02:14:10.306016+0000","last_fullsized":"2024-09-16T02:14:10.306016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.238666+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:10.238666+0000","last_peered":"2024-09-16T02:14:10.238666+0000","last_clean":"2024-09-16T02:14:10.238666+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:10.238666+0000","last_undegraded":"2024-09-16T02:14:10.238666+0000","last_fullsized":"2024-09-16T02:14:10.238666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:16.054 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T02:14:16.054 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-16T02:14:16.054 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-16T02:14:16.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-16T02:14:16.389 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:16.389 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T02:14:16.407 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-16T02:14:16.407 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-16T02:14:16.417 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-16T02:14:16.417 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}} 2024-09-16T02:14:16.417 INFO:tasks.thrashosds:config is {'bluestore_new_sharding': 'random', 'chance_bluestore_reshard': 1, 'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'min_in': 8, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-16T02:14:16.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-16T02:14:16.745 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-16T02:14:16.746 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T02:14:16.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:17.053 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:17.071 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T02:14:15.994531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":445240,"kb_used_data":4416,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504200,"statfs":{"total":1546188226560,"available":1545732300800,"internally_reserved":0,"allocated":4521984,"data_stored":2980230,"data_compressed":334950,"data_compressed_allocated":475136,"data_compressed_original":1810432,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.824582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.396734+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:10.396734+0000","last_peered":"2024-09-16T02:14:10.396734+0000","last_clean":"2024-09-16T02:14:10.396734+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:10.396734+0000","last_undegraded":"2024-09-16T02:14:10.396734+0000","last_fullsized":"2024-09-16T02:14:10.396734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.172783+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:10.172783+0000","last_peered":"2024-09-16T02:14:10.172783+0000","last_clean":"2024-09-16T02:14:10.172783+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:10.172783+0000","last_undegraded":"2024-09-16T02:14:10.172783+0000","last_fullsized":"2024-09-16T02:14:10.172783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.393169+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:10.393169+0000","last_peered":"2024-09-16T02:14:10.393169+0000","last_clean":"2024-09-16T02:14:10.393169+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:10.393169+0000","last_undegraded":"2024-09-16T02:14:10.393169+0000","last_fullsized":"2024-09-16T02:14:10.393169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.237483+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:10.237483+0000","last_peered":"2024-09-16T02:14:10.237483+0000","last_clean":"2024-09-16T02:14:10.237483+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:10.237483+0000","last_undegraded":"2024-09-16T02:14:10.237483+0000","last_fullsized":"2024-09-16T02:14:10.237483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.180900+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:10.180900+0000","last_peered":"2024-09-16T02:14:10.180900+0000","last_clean":"2024-09-16T02:14:10.180900+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:10.180900+0000","last_undegraded":"2024-09-16T02:14:10.180900+0000","last_fullsized":"2024-09-16T02:14:10.180900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.239397+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:10.239397+0000","last_peered":"2024-09-16T02:14:10.239397+0000","last_clean":"2024-09-16T02:14:10.239397+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:10.239397+0000","last_undegraded":"2024-09-16T02:14:10.239397+0000","last_fullsized":"2024-09-16T02:14:10.239397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.179194+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:10.179194+0000","last_peered":"2024-09-16T02:14:10.179194+0000","last_clean":"2024-09-16T02:14:10.179194+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:10.179194+0000","last_undegraded":"2024-09-16T02:14:10.179194+0000","last_fullsized":"2024-09-16T02:14:10.179194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.306016+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:10.306016+0000","last_peered":"2024-09-16T02:14:10.306016+0000","last_clean":"2024-09-16T02:14:10.306016+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:10.306016+0000","last_undegraded":"2024-09-16T02:14:10.306016+0000","last_fullsized":"2024-09-16T02:14:10.306016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.238666+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:10.238666+0000","last_peered":"2024-09-16T02:14:10.238666+0000","last_clean":"2024-09-16T02:14:10.238666+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:10.238666+0000","last_undegraded":"2024-09-16T02:14:10.238666+0000","last_fullsized":"2024-09-16T02:14:10.238666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:17.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:17.382 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:17.382 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:17.401 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T02:14:15.994531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":445240,"kb_used_data":4416,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509504200,"statfs":{"total":1546188226560,"available":1545732300800,"internally_reserved":0,"allocated":4521984,"data_stored":2980230,"data_compressed":334950,"data_compressed_allocated":475136,"data_compressed_original":1810432,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.824582"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.396734+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:10.396734+0000","last_peered":"2024-09-16T02:14:10.396734+0000","last_clean":"2024-09-16T02:14:10.396734+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:10.396734+0000","last_undegraded":"2024-09-16T02:14:10.396734+0000","last_fullsized":"2024-09-16T02:14:10.396734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.172783+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:10.172783+0000","last_peered":"2024-09-16T02:14:10.172783+0000","last_clean":"2024-09-16T02:14:10.172783+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:10.172783+0000","last_undegraded":"2024-09-16T02:14:10.172783+0000","last_fullsized":"2024-09-16T02:14:10.172783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.393169+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:10.393169+0000","last_peered":"2024-09-16T02:14:10.393169+0000","last_clean":"2024-09-16T02:14:10.393169+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:10.393169+0000","last_undegraded":"2024-09-16T02:14:10.393169+0000","last_fullsized":"2024-09-16T02:14:10.393169+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.237483+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:10.237483+0000","last_peered":"2024-09-16T02:14:10.237483+0000","last_clean":"2024-09-16T02:14:10.237483+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:10.237483+0000","last_undegraded":"2024-09-16T02:14:10.237483+0000","last_fullsized":"2024-09-16T02:14:10.237483+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.180900+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:10.180900+0000","last_peered":"2024-09-16T02:14:10.180900+0000","last_clean":"2024-09-16T02:14:10.180900+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:10.180900+0000","last_undegraded":"2024-09-16T02:14:10.180900+0000","last_fullsized":"2024-09-16T02:14:10.180900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.239397+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:10.239397+0000","last_peered":"2024-09-16T02:14:10.239397+0000","last_clean":"2024-09-16T02:14:10.239397+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:10.239397+0000","last_undegraded":"2024-09-16T02:14:10.239397+0000","last_fullsized":"2024-09-16T02:14:10.239397+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.179194+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:10.179194+0000","last_peered":"2024-09-16T02:14:10.179194+0000","last_clean":"2024-09-16T02:14:10.179194+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:10.179194+0000","last_undegraded":"2024-09-16T02:14:10.179194+0000","last_fullsized":"2024-09-16T02:14:10.179194+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":21,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.306016+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:10.306016+0000","last_peered":"2024-09-16T02:14:10.306016+0000","last_clean":"2024-09-16T02:14:10.306016+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:10.306016+0000","last_undegraded":"2024-09-16T02:14:10.306016+0000","last_fullsized":"2024-09-16T02:14:10.306016+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"26'112","reported_seq":154,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-16T02:14:10.238666+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:10.238666+0000","last_peered":"2024-09-16T02:14:10.238666+0000","last_clean":"2024-09-16T02:14:10.238666+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:10.238666+0000","last_undegraded":"2024-09-16T02:14:10.238666+0000","last_fullsized":"2024-09-16T02:14:10.238666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,6,8,9,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313222,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,9,11,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313220,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,6,7,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280516,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":120165,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":335872,"data_stored":649074,"data_compressed":49399,"data_compressed_allocated":65536,"data_compressed_original":561152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[4,5,7,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27304,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":120146,"data_compressed":16868,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,4,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:17.402 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T02:14:17.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 29 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:fsid babbcdda-abe3-4b2f-82b4-da47bc408d34 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-09-16T02:13:51.176653+0000 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-09-16T02:14:15.889874+0000 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 5 2024-09-16T02:14:17.724 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client jewel 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release squid 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.02 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 16 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6808/3190094522,v1:172.21.15.3:6809/3190094522] [v2:172.21.15.3:6810/3190094522,v1:172.21.15.3:6811/3190094522] exists,up d00dcd0d-0b5f-48c9-8d98-adaaabf5283a 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6816/1191817370,v1:172.21.15.118:6817/1191817370] [v2:172.21.15.118:6818/1191817370,v1:172.21.15.118:6819/1191817370] exists,up 56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c 2024-09-16T02:14:17.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/3704126567,v1:172.21.15.177:6817/3704126567] [v2:172.21.15.177:6818/3704126567,v1:172.21.15.177:6819/3704126567] exists,up 6bfe62d5-9c5b-44df-bc4f-cf312837c932 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6826/3376854549,v1:172.21.15.195:6827/3376854549] [v2:172.21.15.195:6828/3376854549,v1:172.21.15.195:6829/3376854549] exists,up fe2b6f49-91ab-4692-ab9c-4b3bda3bf429 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6824/1858329864,v1:172.21.15.3:6825/1858329864] [v2:172.21.15.3:6826/1858329864,v1:172.21.15.3:6827/1858329864] exists,up 399375fc-ce28-40e2-89b1-e0fc2c1f44ec 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/2650650539,v1:172.21.15.118:6801/2650650539] [v2:172.21.15.118:6802/2650650539,v1:172.21.15.118:6803/2650650539] exists,up 3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/47176522,v1:172.21.15.177:6825/47176522] [v2:172.21.15.177:6826/47176522,v1:172.21.15.177:6827/47176522] exists,up ac6efa66-27ed-494c-956d-22b90be7a8fd 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6810/3756385243,v1:172.21.15.195:6811/3756385243] [v2:172.21.15.195:6812/3756385243,v1:172.21.15.195:6813/3756385243] exists,up fb2a81f7-b4f5-4c4c-aff8-801e8b73d192 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6816/3976393740,v1:172.21.15.3:6817/3976393740] [v2:172.21.15.3:6818/3976393740,v1:172.21.15.3:6819/3976393740] exists,up 01aca5fb-932f-4ecb-a9ab-c716a5247373 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6824/2321182247,v1:172.21.15.118:6825/2321182247] [v2:172.21.15.118:6826/2321182247,v1:172.21.15.118:6827/2321182247] exists,up d52efa47-5645-4018-b495-ed116da5f7a9 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/3769110015,v1:172.21.15.177:6809/3769110015] [v2:172.21.15.177:6810/3769110015,v1:172.21.15.177:6811/3769110015] exists,up 99005d53-e62d-4c20-8d3c-541fd20cbeb8 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6818/1553304829,v1:172.21.15.195:6819/1553304829] [v2:172.21.15.195:6820/1553304829,v1:172.21.15.195:6821/1553304829] exists,up f460f1c7-0172-4bd9-875f-b34aff85804f 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6800/3221525577,v1:172.21.15.3:6801/3221525577] [v2:172.21.15.3:6802/3221525577,v1:172.21.15.3:6803/3221525577] exists,up 7bd5fea1-6e94-478a-8a8f-87459b4030a2 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/84283894,v1:172.21.15.118:6809/84283894] [v2:172.21.15.118:6810/84283894,v1:172.21.15.118:6811/84283894] exists,up 89c43816-fffc-42a6-98af-7f215631d595 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] [v2:172.21.15.177:6802/1752008388,v1:172.21.15.177:6803/1752008388] exists,up 33cb3383-2d68-4928-be43-cfc215f6b080 2024-09-16T02:14:17.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] [v2:172.21.15.195:6804/3564981919,v1:172.21.15.195:6805/3564981919] exists,up b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e 2024-09-16T02:14:17.744 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6808/3190094522,v1:172.21.15.3:6809/3190094522] [v2:172.21.15.3:6810/3190094522,v1:172.21.15.3:6811/3190094522] exists,up d00dcd0d-0b5f-48c9-8d98-adaaabf5283a', 'osd.1 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6816/1191817370,v1:172.21.15.118:6817/1191817370] [v2:172.21.15.118:6818/1191817370,v1:172.21.15.118:6819/1191817370] exists,up 56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c', 'osd.2 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6816/3704126567,v1:172.21.15.177:6817/3704126567] [v2:172.21.15.177:6818/3704126567,v1:172.21.15.177:6819/3704126567] exists,up 6bfe62d5-9c5b-44df-bc4f-cf312837c932', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6826/3376854549,v1:172.21.15.195:6827/3376854549] [v2:172.21.15.195:6828/3376854549,v1:172.21.15.195:6829/3376854549] exists,up fe2b6f49-91ab-4692-ab9c-4b3bda3bf429', 'osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6824/1858329864,v1:172.21.15.3:6825/1858329864] [v2:172.21.15.3:6826/1858329864,v1:172.21.15.3:6827/1858329864] exists,up 399375fc-ce28-40e2-89b1-e0fc2c1f44ec', 'osd.5 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6800/2650650539,v1:172.21.15.118:6801/2650650539] [v2:172.21.15.118:6802/2650650539,v1:172.21.15.118:6803/2650650539] exists,up 3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3', 'osd.6 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6824/47176522,v1:172.21.15.177:6825/47176522] [v2:172.21.15.177:6826/47176522,v1:172.21.15.177:6827/47176522] exists,up ac6efa66-27ed-494c-956d-22b90be7a8fd', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6810/3756385243,v1:172.21.15.195:6811/3756385243] [v2:172.21.15.195:6812/3756385243,v1:172.21.15.195:6813/3756385243] exists,up fb2a81f7-b4f5-4c4c-aff8-801e8b73d192', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6816/3976393740,v1:172.21.15.3:6817/3976393740] [v2:172.21.15.3:6818/3976393740,v1:172.21.15.3:6819/3976393740] exists,up 01aca5fb-932f-4ecb-a9ab-c716a5247373', 'osd.9 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6824/2321182247,v1:172.21.15.118:6825/2321182247] [v2:172.21.15.118:6826/2321182247,v1:172.21.15.118:6827/2321182247] exists,up d52efa47-5645-4018-b495-ed116da5f7a9', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6808/3769110015,v1:172.21.15.177:6809/3769110015] [v2:172.21.15.177:6810/3769110015,v1:172.21.15.177:6811/3769110015] exists,up 99005d53-e62d-4c20-8d3c-541fd20cbeb8', 'osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6818/1553304829,v1:172.21.15.195:6819/1553304829] [v2:172.21.15.195:6820/1553304829,v1:172.21.15.195:6821/1553304829] exists,up f460f1c7-0172-4bd9-875f-b34aff85804f', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6800/3221525577,v1:172.21.15.3:6801/3221525577] [v2:172.21.15.3:6802/3221525577,v1:172.21.15.3:6803/3221525577] exists,up 7bd5fea1-6e94-478a-8a8f-87459b4030a2', 'osd.13 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6808/84283894,v1:172.21.15.118:6809/84283894] [v2:172.21.15.118:6810/84283894,v1:172.21.15.118:6811/84283894] exists,up 89c43816-fffc-42a6-98af-7f215631d595', 'osd.14 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] [v2:172.21.15.177:6802/1752008388,v1:172.21.15.177:6803/1752008388] exists,up 33cb3383-2d68-4928-be43-cfc215f6b080', 'osd.15 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] [v2:172.21.15.195:6804/3564981919,v1:172.21.15.195:6805/3564981919] exists,up b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e'] 2024-09-16T02:14:17.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-16T02:14:17.867 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:17.867 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mon.a", 2024-09-16T02:14:17.867 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster": "ceph", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket_mode": "", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_client_required": "cephx, none", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_cluster_required": "cephx", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_debug": "true", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_required": "cephx", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_supported": "cephx", 2024-09-16T02:14:17.868 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio": "true", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_poll_ms": "250", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_reap_max": "16", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_async_discard_threads": "0", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_block_size": "4096", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio": "false", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_inflight_ios": "false", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_enable_discard": "false", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry": "3", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-16T02:14:17.869 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash": "0", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring": "false", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_hipri": "false", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_type": "", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_alloc_size": "1048576", 2024-09-16T02:14:17.870 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_allocator": "hybrid", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_buffered_io": "true", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_for_zeros": "false", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_compact_log_sync": "false", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_log_runway": "4194304", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_prefetch": "1048576", 2024-09-16T02:14:17.871 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_flush_size": "524288", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_log_runway": "1048576", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery": "false", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_sync_write": "false", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocation_from_file": "true", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocator": "hybrid", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_async_db_compaction": "true", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-16T02:14:17.872 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi": "true", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_create": "true", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_create": "false", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_path": "", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_size": "0", 2024-09-16T02:14:17.873 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_path": "", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_preallocate_file": "false", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_size": "107374182400", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_create": "false", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_path": "", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_size": "100663296", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs": "true", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-16T02:14:17.874 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune": "true", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size": "0", 2024-09-16T02:14:17.875 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_type": "2q", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_clone_cow": "true", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-16T02:14:17.876 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_mode": "none", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_csum_type": "crc32c", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_freelist": "false", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-16T02:14:17.877 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_small_allocations": "0", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_read": "true", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_write": "false", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-16T02:14:17.878 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fail_eio": "false", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-16T02:14:17.879 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-16T02:14:17.880 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_ignore_data_csum": "false", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_op_age": "5.000000", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_alloc_size": "0", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size": "0", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_deferred_txc": "32", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size": "0", 2024-09-16T02:14:17.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_nid_prealloc": "1024", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_retry_disk_reads": "3", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cf": "true", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-16T02:14:17.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_mem": "512", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_tracing": "false", 2024-09-16T02:14:17.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2": "false", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_write_v2_random": "false", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_zero_block_detection": "false", 2024-09-16T02:14:17.884 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-16T02:14:17.885 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_signatures": "false", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_version": "2", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_signatures": "false", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_version": "2", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_signatures": "false", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_version": "2", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_sign_messages": "true", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "chdir": "", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "client_acl_type": "", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "client_asio_thread_count": "2", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_mid": "0.750000", 2024-09-16T02:14:17.886 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_size": "16384", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_caps_release_delay": "5", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_check_pool_perm": "true", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_force_sync_read": "false", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_getattr_caps": "false", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_features": "", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_tick_delay": "0", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_remount": "false", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_dirsize_rbytes": "true", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_force_lazyio": "false", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_fs": "", 2024-09-16T02:14:17.887 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_release_failure": "false", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_inline_size": "4096", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mds_namespace": "", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_metadata": "", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_gid": "-1", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_timeout": "300", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_uid": "-1", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mountpoint": "/", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_notify_timeout": "10", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc": "true", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty": "104857600", 2024-09-16T02:14:17.888 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_objects": "1000", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_size": "209715200", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_target_dirty": "8388608", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_permissions": "true", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota": "true", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota_df": "true", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_bytes": "0", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_periods": "4", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_min": "131072", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_reconnect_stale": "false", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_shutdown_timeout": "30", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_snapdir": ".snap", 2024-09-16T02:14:17.889 INFO:teuthology.orchestra.run.smithi003.stdout: "client_tick_interval": "1", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_trace": "", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_try_dentry_invalidate": "false", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_faked_inos": "false", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_random_mds": "false", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog": "false", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_port": "12201", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_monitors": "default=true", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog": "false", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_level": "info", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "-", 2024-09-16T02:14:17.890 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network": "", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network_interface": "", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_isal": "false", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_level": "5", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_winsize": "-15", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zstd_level": "1", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_op_num_threads": "6", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_stat_interval": "0", 2024-09-16T02:14:17.891 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_cpu_cores": "", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_num_threads": "0", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook": "", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook_timeout": "10", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "daemonize": "false", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "daos_pool": "tank", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_alienstore": "0/5", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_allow_any_pool_priority": "false", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok": "1/5", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok_assert_abort": "false", 2024-09-16T02:14:17.892 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asserts_on_shutdown": "true", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_auth": "20/20", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bdev": "1/3", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluefs": "1/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluestore": "1/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_buffer": "0/1", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_dedup": "0/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_exporter": "1/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephfs_mirror": "0/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephsqlite": "0/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_client": "0/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_compressor": "1/5", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_context": "0/1", 2024-09-16T02:14:17.893 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crimson_interrupt": "0/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crush": "1/1", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crypto": "1/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cyanstore": "0/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_deliberately_leak_memory": "false", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_disable_randomized_ping": "false", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_dpdk": "1/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_eventtrace": "1/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filer": "0/1", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filestore": "1/3", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_finisher": "1/1", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_fuse": "1/5", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeat_testing_span": "0", 2024-09-16T02:14:17.894 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeatmap": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_javaclient": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journal": "1/3", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journaler": "0/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_kstore": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_lockdep": "0/1", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mclock": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_balancer": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_locker": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log_expire": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_migrator": "1/5", 2024-09-16T02:14:17.895 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_quiesce": "3/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_memstore": "1/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgr": "2/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgrc": "1/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mon": "20/20", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_monc": "0/10", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ms": "1/1", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_none": "0/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objclass": "0/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objectcacher": "0/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objecter": "0/1", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_optracker": "0/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_osd": "1/5", 2024-09-16T02:14:17.896 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_paxos": "20/20", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_perfcounter": "1/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_prioritycache": "1/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rados": "0/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd": "0/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_mirror": "0/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_pwl": "0/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_replay": "0/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_refs": "0/0", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_reserver": "1/1", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw": "1/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_access": "1/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_datacache": "1/5", 2024-09-16T02:14:17.897 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_dbstore": "1/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_flight": "1/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_notification": "1/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_sync": "1/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rocksdb": "4/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_backref": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cache": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cleaner": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_device": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_epm": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-16T02:14:17.898 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_journal": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_lba": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_odata": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_omap": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_onode": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_t": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_tm": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_striper": "0/1", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_test": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_throttle": "1/1", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_timer": "0/1", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_tp": "0/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_trace": "1/5", 2024-09-16T02:14:17.899 INFO:teuthology.orchestra.run.smithi003.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "device_failure_prediction_mode": "none", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_graylog": "false", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_journald": "false", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_stderr": "true", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_syslog": "false", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event_tracing": "false", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_addr": "0.0.0.0", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_cert_file": "", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_http_port": "9926", 2024-09-16T02:14:17.900 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_key_file": "", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_prio_limit": "5", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sort_metrics": "true", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_stats_period": "5", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "fake_statfs_for_testing": "0", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "fatal_signal_handlers": "true", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_purge_ops": "10", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_truncate_ops": "128", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_apply_finisher_threads": "1", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_blackhole": "false", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_clone_range": "true", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_snap": "true", 2024-09-16T02:14:17.901 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_caller_concurrency": "10", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_collect_device_partition_information": "true", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_commit_timeout": "600.000000", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_inject_read_err": "false", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_omap_check": "false", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_verify_split": "false", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_dump_file": "", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fadvise": "true", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fail_eio": "true", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_shards": "16", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_size": "128", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap": "false", 2024-09-16T02:14:17.902 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap_threshold": "4096", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_inject_stall": "0", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_parallel": "false", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_trailing": "false", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_writeahead": "false", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_kill_at": "0", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-16T02:14:17.903 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs": "0", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size": "0", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_merge_threshold": "-10", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_odsync_write": "false", 2024-09-16T02:14:17.904 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend": "rocksdb", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend_path": "", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_timeout": "60", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_threads": "2", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_punch_hole": "false", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-16T02:14:17.905 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_ops": "50", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_seek_data_hole": "false", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc": "false", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_splice": "false", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_multiple": "2", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_rand_factor": "20", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_update_to": "1000", 2024-09-16T02:14:17.906 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_enable": "true", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-16T02:14:17.907 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_xfs_extsize": "false", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_zfs_snap": "false", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fio_dir": "/tmp/fio", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "babbcdda-abe3-4b2f-82b4-da47bc408d34", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_allow_other": "true", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_atomic_o_trunc": "true", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_big_writes": "true", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_debug": "false", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_default_permissions": "false", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_disable_pagecache": "false", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_max_write": "0", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_multithreaded": "true", 2024-09-16T02:14:17.908 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_require_active_mds": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_set_user_groups": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_move": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_read": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_write": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_use_invalidate_cb": "true", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_target_name": "ceph", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_file": "", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_inject_failure": "0", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_interval": "5", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "host": "smithi003", 2024-09-16T02:14:17.909 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-16T02:14:17.910 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "inject_early_sigterm": "false", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_agent_port": "6799", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_tracing_enable": "false", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_aio": "true", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_align_min_size": "65536", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_align": "true", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_size": "4096", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_dio": "true", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_discard": "false", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_force_aio": "false", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_ignore_corruption": "false", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_bytes": "10485760", 2024-09-16T02:14:17.911 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_entries": "100", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_replay_from": "0", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_write_header_frequency": "0", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_zero_on_create": "false", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prefetch_periods": "10", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prezero_periods": "5", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_write_head_interval": "15", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "keyfile": "", 2024-09-16T02:14:17.912 INFO:teuthology.orchestra.run.smithi003.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_backend": "rocksdb", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_default_stripe_size": "65536", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount": "false", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_bytes": "67108864", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_ops": "512", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_nid_prealloc": "1024", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_onode_map_size": "1024", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_submit_transaction": "false", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_transaction": "false", 2024-09-16T02:14:17.913 INFO:teuthology.orchestra.run.smithi003.stdout: "librados_thread_count": "2", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep": "false", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep_force_backtrace": "false", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_coarse_timestamps": "true", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_flush_on_exit": "false", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_host": "127.0.0.1", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_port": "12201", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_new": "1000", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_recent": "10000", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stderr_prefix": "", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stop_at_utilization": "0.970000", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": "true", 2024-09-16T02:14:17.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_graylog": "false", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_journald": "false", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_stderr": "false", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_syslog": "false", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "max_rotating_auth_attempts": "10", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_action_on_write_error": "1", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_alternate_name_max": "8192", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_asio_thread_count": "2", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_export_pin": "true", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_dirs": "true", 2024-09-16T02:14:17.915 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_interval": "5", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_interval": "10", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max": "-1", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max_until": "-1", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_merge_size": "50", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_midchunk": "0.300000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_start": "0.200000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_minchunk": "0.001000", 2024-09-16T02:14:17.916 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_mode": "0", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_max": "1.200000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_min": "0.800000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_overload_epochs": "2", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_bits": "3", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_size": "10000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_target_decay": "10.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-16T02:14:17.917 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_grace": "15.000000", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_interval": "4.000000", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_mid": "0.700000", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_delay": "0", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-16T02:14:17.918 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_release_free_interval": "10", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_reservation": "0.050000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_interval": "1", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_threshold": "262144", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_prealloc_inos": "1000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_connect_bootstrapping": "false", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_damage_table_max_entries": "10000", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-16T02:14:17.919 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_auth_pins": "false", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_frag": "false", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_scatterstat": "false", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_subtrees": "false", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_decay_halflife": "5.000000", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_default_dir_hash": "2", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_defer_session_stale": "true", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_deny_all_reconnect": "false", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_keys_per_op": "16384", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_commit_size": "10", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_entries": "0", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_prefetch": "true", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-16T02:14:17.920 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_on_map": "false", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_early_reply": "true", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enable_op_tracker": "true", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enforce_unique_name": "true", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random": "true", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-16T02:14:17.921 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_summarize_threshold": "10", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_health_dummy": "false", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_migrator_session_race": "false", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-16T02:14:17.922 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_join_fs": "", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_journal_format": "1", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_create_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_dirfrag_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_export_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_import_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_expire_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_replay_at": "0", 2024-09-16T02:14:17.923 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_link_at": "0", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_mdstable_at": "0", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_openc_at": "0", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_rename_at": "0", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_shutdown_at": "0", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_event_large_threshold": "524288", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_events_per_segment": "1024", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_events": "-1", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_segments": "128", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_pause": "false", 2024-09-16T02:14:17.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_segment_size": "0", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_threshold": "128", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_warn_factor": "2.000000", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_caps_per_client": "1048576", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_flushes": "100000", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_requests": "100000", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_export_size": "20971520", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_file_recover": "32", 2024-09-16T02:14:17.925 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_files": "64", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops": "8192", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_snaps_per_dir": "100", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_metrics_update_interval": "2", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_per_client": "100", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_working_set": "10000", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_numa_node": "-1", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_complaint_time": "30.000000", 2024-09-16T02:14:17.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_duration": "600", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_size": "20", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_size": "20", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_log_threshold": "5", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_grace": "15", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_interval": "5", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_caps": "30000", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-16T02:14:17.927 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_threshold": "262144", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_interval": "1.000000", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_gid": "0", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_uid": "0", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_scrub_stats_review_period": "1", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-16T02:14:17.928 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_shutdown_check": "0", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_skip_ino": "0", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-16T02:14:17.929 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_max_uid": "4294967294", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_min_uid": "0", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_rstat": "false", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_standby_replay_damaged": "false", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_symlink_recovery": "true", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_exports": "0", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_fragments": "0", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_tick_interval": "5.000000", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_valgrind_exit": "false", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_backtrace": "1", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_scatter": "false", 2024-09-16T02:14:17.930 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_sessions": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool_debug": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_device_bytes": "1073741824", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_set": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_size": "65536", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_bytes": "134217728", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_messages": "512", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-16T02:14:17.931 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_disabled_modules": "", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_enable_op_tracker": "true", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_creating": "1024", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_num_change": "128", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_bytes": "134217728", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_messages": "128", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_bytes": "134217728", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_messages": "128", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_duration": "600", 2024-09-16T02:14:17.932 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_size": "20", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_op_log_threshold": "5", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_bytes": "536870912", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_messages": "8192", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_pool": "true", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_standby_modules": "true", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_period": "5", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_threshold": "5", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_tick_period": "2", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-16T02:14:17.933 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_delete": "true", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_size_one": "true", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_auth_validate_all_caps": "true", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_bytes": "104857600", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_directed_command_retry": "5", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_parallel": "3", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_log_interval": "1.000000", 2024-09-16T02:14:17.934 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_interval": "10.000000", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_target_rank": "-1", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_level": "debug", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_file": "true", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-16T02:14:17.935 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_journald": "false", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_bootstrap": "false", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_start": "false", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_trim": "true", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cpu_threads": "4", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_crush_min_required_version": "hammer", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_daemon_bytes": "419430400", 2024-09-16T02:14:17.936 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_crit": "5", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_warn": "5", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_size_warn": "16106127360", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_json": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_transactions": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_extra_checks": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_reef": "false", 2024-09-16T02:14:17.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_squid": "false", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_mkfs_grace": "120", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_uptime_grace": "60", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_default_strategy": "1", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_timeout": "5.000000", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_divisor": "2", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_enable_op_tracker": "true", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fake_pool_delete": "false", 2024-09-16T02:14:17.938 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_force_quorum_join": "false", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_globalid_prealloc": "10000", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_detail_to_clog": "false", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_log_update_period": "5", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_max_detail": "50", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog": "true", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_interval": "600", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host": "172.21.15.3,172.21.15.118,172.21.15.177", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host_override": "", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_initial_members": "", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-16T02:14:17.939 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease": "5.000000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_full_interval": "50", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max": "10000", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max_summary": "50", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_epochs": "500", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-16T02:14:17.940 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_osd": "10000", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pg_per_osd": "10000", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pool_pg_num": "65536", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_force_trim_to": "0", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_skip_sanity": "false", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_autotune": "true", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_target": "2147483648", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_beacon_grace": "30", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_digest_period": "5", 2024-09-16T02:14:17.941 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_inactive_grace": "60", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_min_osdmap_epochs": "500", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_complaint_time": "30", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_duration": "600", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_size": "20", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_size": "20", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_log_threshold": "5", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-16T02:14:17.942 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_in": "false", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size": "500", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_interval": "600", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-16T02:14:17.943 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_force_trim_to": "0", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_down_reporters": "2", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-16T02:14:17.944 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_report_timeout": "900", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-16T02:14:17.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_stuck_threshold": "60", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_probe_timeout": "2.000000", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_change": "0.050000", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_osds": "4", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-16T02:14:17.946 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_interval": "300", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_max_keys": "100", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_timeout": "300", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_session_timeout": "300", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_smart_report_timeout": "5", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stat_smooth_intervals": "6", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_min_size": "2", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_size": "4", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-16T02:14:17.947 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_debug": "false", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_provider_kill_at": "0", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_requester_kill_at": "0", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_timeout": "60.000000", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_target_pg_per_osd": "100", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_tick_interval": "5", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_interval": "300.000000", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_use_min_delay_socket": "false", 2024-09-16T02:14:17.948 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_older_version_delay": "604800", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_misplaced": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_older_version": "true", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-16T02:14:17.949 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": "", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-16T02:14:17.950 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_tracing_enabled": "false", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_op_threads": "3", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_cm": "false", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_device_name": "", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_dscp": "96", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_local_gid": "", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_port_num": "1", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-16T02:14:17.951 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_sl": "3", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_support_srq": "true", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_type": "ib", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_reap_threshold": "5", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_before_connect": "false", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv4": "true", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv6": "false", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr1": "true", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr2": "true", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_max": "7568", 2024-09-16T02:14:17.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_min": "6800", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_count": "3", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_delay": "5", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_client": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mds": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mgr": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mon": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_osd": "false", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_mode": "crc secure", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_mode": "crc secure", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_type": "", 2024-09-16T02:14:17.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_compress_secure": "false", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_idle_timeout": "900", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_ready_timeout": "10", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_data": "true", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_header": "true", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bad_msg": "false", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bug": "true", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_old_message": "true", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_skipped_message": "false", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_coremask": "0xF", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-16T02:14:17.954 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_enable_tso": "true", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hugepages": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_lro": "true", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_memory_channel": "4", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_pmd": "", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_port_id": "0", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-16T02:14:17.955 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_on_send": "false", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_initial_backoff": "0.200000", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_max": "1.000000", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_type": "osd", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_network_congestion": "0", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_socket_failures": "2500", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_learn_addr_from_peer": "true", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_accept_failures": "4", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_backoff": "15.000000", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_client_mode": "secure crc", 2024-09-16T02:14:17.956 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_service_mode": "secure crc", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_min_size": "1024", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_mode": "none", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_min_cost": "65536", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_public_type": "", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_service_mode": "crc secure", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_listen_backlog": "512", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_nodelay": "true", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_rcvbuf": "0", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-16T02:14:17.957 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_type": "async+posix", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "no_config_file": "false", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_completion_locks_per_session": "32", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_ops": "1024", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_tick_interval": "5.000000", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_timeout": "10.000000", 2024-09-16T02:14:17.958 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_blackhole": "false", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "openssl_engine_opts": "", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_delay_time": "5.000000", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_hist_halflife": "1000", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_low_ops": "2", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_ops": "4", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_slop": "0.020000", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_asio_thread_count": "2", 2024-09-16T02:14:17.959 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_async_recovery_min_cost": "100", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_max": "512", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_min": "64", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_degraded": "false", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_peering": "false", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_unfound": "true", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_beacon_report_interval": "300", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_duration": "30", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_max_block_size": "67108864", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-16T02:14:17.960 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blkin_trace_all": "false", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_for_log_corruption": "false", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_update_on_start": "true", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_cap": "256", 2024-09-16T02:14:17.961 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_size_cap": "524288000", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_op_priority": "63", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_watch_timeout": "30", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_command_max_records": "256", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_compact_on_start": "false", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_on_start": "true", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_weight_set": "true", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-16T02:14:17.962 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_feed_pullee": "-1", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_misdirected_ops": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_acting_change": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_purge_strays": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_op_order": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-16T02:14:17.963 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_shutdown": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_trim_objects": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_snaps": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_keys": "1024", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-16T02:14:17.964 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_stride": "524288", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_notify_timeout": "30", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep": "0.000000", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_discard_disconnected_ops": "true", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ec_partial_reads": "true", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_enable_op_tracker": "true", 2024-09-16T02:14:17.965 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_info": "true", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown": "true", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_function_tracing": "false", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_grace": "20", 2024-09-16T02:14:17.966 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_interval": "6", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_peers": "10", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_size": "2000", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_stale": "600", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_max_size": "100000", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_min_size": "1000", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-16T02:14:17.967 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_size": "5120", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_kill_backfill_at": "0", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_cache_size": "50", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_dedup": "true", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max": "40", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_share_max_epochs": "40", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_name_len": "100", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_size": "0", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_backfills": "1", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_count": "5", 2024-09-16T02:14:17.968 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_period": "600", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_name_len": "2048", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_namespace_len": "256", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_size": "134217728", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_blocked_by": "16", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_log_entries": "10000", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pgls": "1024", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_cost": "8388608", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_objects": "10", 2024-09-16T02:14:17.969 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_scrubs": "3", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_trimming_pgs": "2", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_op_reply_len": "64", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_size": "90", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-16T02:14:17.970 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_profile": "balanced", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_base": "805306368", 2024-09-16T02:14:17.971 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_min": "134217728", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target": "4294967296", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_autotune": "false", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_pg_log_entries": "250", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_recovery_priority": "0", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_interval": "5", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-16T02:14:17.972 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_cache_shards": "32", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_op_tracker_shard": "32", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_auto_affinity": "true", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_node": "-1", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_prefer_iface": "true", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objecter_finishers": "1", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore": "bluestore", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_fuse": "false", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_tracing": "false", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_complaint_time": "30.000000", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_duration": "600", 2024-09-16T02:14:17.973 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_size": "20", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_size": "20", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_log_threshold": "5", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards": "0", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_hdd": "5", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_ssd": "8", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_min_cost": "65536", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-16T02:14:17.974 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue_cut_off": "high", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_timeout": "15", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_open_classes_on_start": "true", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_os_flags": "0", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_peering_op_priority": "255", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_cost": "1048576", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_priority": "5", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_max": "10000", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_min": "100", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-16T02:14:17.975 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crimson": "false", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-16T02:14:17.976 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flags": "0", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_min_size": "0", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_num": "32", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pgp_num": "0", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_ratio": "70", 2024-09-16T02:14:17.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_size": "2", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_type": "replicated", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_push_per_object_cost": "1000", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap": "true", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_cost": "20971520", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active": "0", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-16T02:14:17.978 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_single_start": "1", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_priority": "3", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_priority": "5", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep": "0.000000", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_repair_during_recovery": "false", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_requested_scrub_priority": "120", 2024-09-16T02:14:17.979 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair": "false", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_hour": "0", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_week_day": "0", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_max": "15", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_min": "5", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_cost": "52428800", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_during_recovery": "false", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_hour": "0", 2024-09-16T02:14:17.980 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_week_day": "0", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_event_cost": "4096", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_invalid_stats": "true", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_preemptions": "5", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_priority": "5", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_retry_delay": "30", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-16T02:14:17.981 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_retry_trimming": "10", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_sleep": "0.000000", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_data_digest": "false", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_smart_report_timeout": "5", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_cost": "1048576", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_priority": "5", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-16T02:14:17.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_transaction_size": "30", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-16T02:14:17.983 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tracing": "false", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_use_stale_snap": "false", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "osdc_blkin_trace_all": "false", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_kill_at": "0", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_max_join_drift": "10", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min": "500", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min_wait": "0.050000", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_propose_interval": "1.000000", 2024-09-16T02:14:17.984 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max": "500", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_min": "250", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_stash_full_interval": "25", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_max": "500", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_min": "250", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "perf": "true", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "-", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrv": "", 2024-09-16T02:14:17.985 INFO:teuthology.orchestra.run.smithi003.stdout: "public_bind_addr": "-", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network": "", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network_interface": "", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_busy_polling": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_enabled": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_session_max_number": "256", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_mon_op_timeout": "0", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_osd_op_timeout": "0", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_tracing": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_atime_update_interval": "60", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_parent_reads": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_snap_reads": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blkin_trace_all": "false", 2024-09-16T02:14:17.986 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache": "true", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_policy": "writearound", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_size": "33554432", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_clone_copy_on_read": "false", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_compression_hint": "none", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_concurrent_management_ops": "10", 2024-09-16T02:14:17.987 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_clone_format": "auto", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_data_pool": "", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_features": "61", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_format": "2", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_map_options": "", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_order": "22", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_pool": "rbd", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_count": "0", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_unit": "0", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-16T02:14:17.988 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_enable_alloc_hint": "true", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler": "simple", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-16T02:14:17.989 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_order": "24", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_pool": "", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_splay_width": "4", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_parent_reads": "false", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_snap_reads": "false", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-16T02:14:17.990 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-16T02:14:17.991 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mtime_update_interval": "60", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_non_blocking_aio": "true", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_thread_timeout": "60", 2024-09-16T02:14:17.992 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_threads": "1", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_parent_cache_enabled": "false", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_plugins": "", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst": "0", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_limit": "0", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_exclude_ops": "0", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst": "0", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_limit": "0", 2024-09-16T02:14:17.993 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-16T02:14:17.994 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_read_from_replica_policy": "default", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_skip_partial_discard": "true", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_tracing": "false", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_names": "true", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_pool": "true", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_base_url": "", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_log_level": "", 2024-09-16T02:14:17.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_acl_grants_max_num": "100", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_admin_entry": "admin", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_asio_assert_yielding": "false", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_backend_store": "rados", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_barbican_url": "", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_beast_enable_async": "true", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache": "false", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-16T02:14:17.996 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_enabled": "true", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_expiry_interval": "900", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_lru_size": "25000", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_config_store": "rados", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_content_length_compat": "false", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress": "true", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-16T02:14:17.997 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cors_rules_max_num": "100", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cross_domain_policy": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_addr": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_password": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_username": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_require_ssl": "true", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-16T02:14:17.998 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-16T02:14:17.999 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_addr": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_auth": "token", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_namespace": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_prefix": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_token_file": "", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_buffersize": "524288", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_time": "300", 2024-09-16T02:14:18.000 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-16T02:14:18.001 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_changes_size": "1000", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_num_shards": "128", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_window": "30", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-16T02:14:18.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_disable_s3select": "false", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-16T02:14:18.003 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_name": "", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_s3website_name": "", 2024-09-16T02:14:18.004 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_gc_threads": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_lc_threads": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_mdsearch": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_ops_log": "false", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_quota_threads": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_static_website": "false", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_usage_log": "false", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enforce_swift_acls": "true", 2024-09-16T02:14:18.005 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_exit_timeout_secs": "120", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_expose_bucket": "false", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_extended_http_attrs": "", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_filter": "none", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontends": "beast port=7480", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred": "50", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_objs": "32", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-16T02:14:18.006 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_period": "3600", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_init_timeout": "300", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_connection_idle": "300", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-16T02:14:18.007 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_domain": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password_path": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_project": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_tenant": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token_path": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_user": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_domain": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_password": "", 2024-09-16T02:14:18.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_project": "", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_user": "", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_url": "", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_debug_interval": "-1", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_lock_max_time": "90", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_objs": "32", 2024-09-16T02:14:18.009 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_rules": "1000", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_worker": "3", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_thread_delay": "0", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_dnattr": "uid", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchfilter": "", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_uri": "", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-16T02:14:18.010 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_http_headers": "", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name_utc": "false", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_name_len": "0", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_size": "0", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_chunk_size": "4194304", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-16T02:14:18.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_listing_results": "5000", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_notify_retries": "10", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_param_size": "1048576", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_size": "5368709120", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_slo_entries": "1000", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_log_max_shards": "64", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mp_lock_max_time": "600", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-16T02:14:18.012 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_max_gc": "300", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-16T02:14:18.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_async_rados_threads": "32", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_control_oids": "8", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_numa_node": "-1", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_chunk_size": "100", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_gc_interval": "600", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_timeout": "600", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_tracing": "false", 2024-09-16T02:14:18.014 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_token": "", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_url": "", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_verify_ssl": "true", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_rados": "false", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_socket_path": "", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval": "2.000000", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-16T02:14:18.015 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lanes": "3", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_partitions": "3", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_continue": "true", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-16T02:14:18.016 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_tracing": "false", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm": "", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_id": "", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region": "", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_request_uri": "", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_batch_size": "64", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-16T02:14:18.017 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_max_aio": "128", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_num_logs": "16", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_thread_interval": "600", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_resolve_cname": "false", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_run_sync_thread": "true", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-16T02:14:18.018 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_scheduler_type": "throttler", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_script_uri": "", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_service_provider_name": "", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_id": "", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_secret": "", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_entry": "sts", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_key": "sts", 2024-09-16T02:14:18.019 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_min_session_duration": "900", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_token_introspection_url": "", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_account_in_url": "false", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_entry": "auth", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_url": "", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_custom_header": "", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_need_stats": "true", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_tenant_name": "", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_token_expiration": "86400", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url": "", 2024-09-16T02:14:18.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url_prefix": "swift", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_lease_period": "120", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-16T02:14:18.021 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_thread_pool_size": "512", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_comment": "", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_createby": "", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_encoding": "", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_flag": "false", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_origin": "", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-16T02:14:18.022 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_tracker": "", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_trust_forwarded_https": "false", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_shards": "32", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_user_shards": "1", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_use_opa_authz": "false", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache": "false", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_max_buckets": "1000", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_policies_max_num": "100", 2024-09-16T02:14:18.023 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_unique_email": "true", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_verify_ssl": "true", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone": "", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_id": "", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup": "", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_id": "", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-16T02:14:18.024 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_auto_session_clear": "true", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_block_size": "4096", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_size": "536870912", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-16T02:14:18.025 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_index_type": "binary_search", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_partition_filters": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_perf": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-16T02:14:18.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "run_dir": "/var/run/ceph", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_block_create": "true", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cache_lru_size": "67108864", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_max_object_size": "16777216", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_device_size": "53687091200", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_full_integrity_check": "false", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_capacity": "16", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-16T02:14:18.027 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_main_device_type": "SSD", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_segment_size": "67108864", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "set_keepcaps": "false", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "setgroup": "", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser": "", 2024-09-16T02:14:18.028 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser_match_path": "", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "thp": "false", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_default_timeout": "60", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "throttler_perf_counter": "true", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_dir": "/tmp", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "uadk_compressor_enabled": "false", 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-16T02:14:18.029 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:18.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-16T02:14:18.188 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-09-16T02:14:18.188 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:14:18.206 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-09-16T02:14:18.206 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:14:18.225 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-09-16T02:14:18.225 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:14:18.243 DEBUG:teuthology.orchestra.run.smithi177:> type ceph-objectstore-tool 2024-09-16T02:14:18.248 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T02:14:18.248 DEBUG:teuthology.orchestra.run.smithi118:> type ceph-objectstore-tool 2024-09-16T02:14:18.254 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T02:14:18.254 DEBUG:teuthology.orchestra.run.smithi195:> type ceph-objectstore-tool 2024-09-16T02:14:18.260 INFO:teuthology.orchestra.run.smithi195.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T02:14:18.261 DEBUG:teuthology.orchestra.run.smithi003:> type ceph-objectstore-tool 2024-09-16T02:14:18.265 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T02:14:18.265 INFO:teuthology.run_tasks:Running task rados... 2024-09-16T02:14:18.275 INFO:tasks.rados:Beginning rados... 2024-09-16T02:14:18.275 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-16T02:14:18.275 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_object_clean_region_max_num_intervals': 1000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.1, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'slow request', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}} 2024-09-16T02:14:18.275 INFO:tasks.rados:config is {'clients': ['client.0'], 'ec_pool': True, 'erasure_code_profile': {'c': 2, 'crush-failure-domain': 'osd', 'k': 4, 'm': 3, 'name': 'shecprofile', 'plugin': 'shec'}, 'objects': 50, 'op_weights': {'append': 100, 'copy_from': 50, 'delete': 50, 'read': 100, 'rmattr': 25, 'rollback': 50, 'setattr': 25, 'snap_create': 50, 'snap_remove': 50, 'write': 0}, 'ops': 400, 'write_append_excl': False} 2024-09-16T02:14:18.276 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-16T02:14:18.283 INFO:tasks.rados:joining rados 2024-09-16T02:14:18.283 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-16T02:14:18.283 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:14:18.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T02:14:18.284 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-16T02:14:18.284 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-16T02:14:18.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:18.285 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-16T02:14:18.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:18.285 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-16T02:14:18.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:14:18.286 INFO:tasks.rados:clients are ['client.0'] 2024-09-16T02:14:18.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set shecprofile c=2 crush-failure-domain=osd k=4 m=3 name=shecprofile plugin=shec 2024-09-16T02:14:18.307 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:18.302+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:18.386 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:18.378+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:18.430 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:18.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:18.430 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:18.430 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:18.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:18.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:18.464 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:18.487 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:18.484+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:18.504 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:18.504 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:18.532 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:18.532 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:18.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:18.573 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:18.587 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:18.582+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:18.597 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:18.610 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:18.610 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:18.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:18.650 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:18.650 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:18.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:18.690 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:18.690 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:18.682+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:18.729 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:18.729 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:18.751 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:18.751 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:18.760 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:18.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:18.777 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:18.777 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:18.804 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:18.804 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:18.805 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:18.798+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:18.832 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:18.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:18.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:18.859 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:18.891 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:18.886+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:18.895 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:18.895 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:18.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:18.890+0000 7f8fd8907640 0 -- 172.21.15.3:0/2650659335 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f8fb002ded0 msgr2=0x7f8fb002e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:18.898 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:18.898 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:18.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:18.992 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:18.992+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:19.074 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.074 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:19.093 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:19.091+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:19.131 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:19.131 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:19.163 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:19.164 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:19.194 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:19.192+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:19.206 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:14:19.206 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T02:14:19.207 INFO:teuthology.orchestra.run.smithi003.stderr:load: shec 2024-09-16T02:14:19.219 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.219 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T02:14:19.242 INFO:tasks.rados:starting run 0 out of 1 2024-09-16T02:14:19.242 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-16T02:14:19.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure shecprofile 2024-09-16T02:14:19.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:19.295 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:19.292+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:19.397 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:19.392+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:19.468 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.468 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:19.498 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:19.491+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:19.598 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:19.594+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:19.599 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.599 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:19.699 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:19.696+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:19.767 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.767 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:19.800 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:19.795+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:19.900 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:19.901 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:19.901 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:19.900+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:19.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:14:20.001 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:19.995+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:20.041 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.041 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:14:20.102 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:20.099+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:20.175 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.176 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.190 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:14:20.202 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:20.196+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:20.222 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T02:14:20.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T02:14:20.303 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:20.299+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:20.324 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.324 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:14:20.404 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:20.400+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:20.501 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.501 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.505 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:20.504+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:20.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:14:20.607 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:20.604+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:20.632 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.632 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:14:20.708 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:20.704+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:20.766 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.766 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:14:20.809 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:20.806+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:20.904 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:20.904 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:20.910 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:20.907+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:20.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:14:21.011 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:21.008+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:21.083 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.083 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:14:21.112 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:21.112+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:21.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:21.213 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:21.212+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:21.216 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.216 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:14:21.238 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' created 2024-09-16T02:14:21.240 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T02:14:21.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:14:21.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T02:14:21.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-16T02:14:21.314 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:21.312+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:21.360 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.360 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:14:21.389 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:21.389 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:21.415 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:21.412+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:21.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:21.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:21.448 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:21.448 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:21.490 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:21.497 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.497 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:14:21.511 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:21.512 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:21.516 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:21.512+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:21.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:21.548 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:21.587 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:21.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:21.616 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:21.612+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:21.627 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:21.627 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:21.631 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.631 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:14:21.664 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:21.664 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:21.685 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:21.686 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:21.710 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:21.710 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:21.717 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:21.715+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:21.736 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:21.736 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:21.768 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:21.768 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:21.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:21.770 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:21.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:14:21.795 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:21.795 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:21.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:21.798+0000 7fa5d259c640 0 -- 172.21.15.3:0/2591382244 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fa5d42240b0 msgr2=0x7fa5d42264a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:21.826 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:21.827+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:21.829 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:21.829 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:21.860 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:21.860 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:21.920 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:21.916+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.880947932, 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:14:21.956 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:14:21.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:14:21.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:48 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.963 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575787+0000", 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.376422717000001, 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014150965, 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 48 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575787+0000", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575787+0000", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575788+0000", 2024-09-16T02:14:21.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9999999999999996e-07 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575795+0000", 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4960000000000001e-06 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575805+0000", 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0239e-05 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575813+0000", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7559999999999999e-06 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.586904+0000", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011090933000000001 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.587128+0000", 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224613 2024-09-16T02:14:21.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.587552+0000", 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042333399999999998 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.589473+0000", 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001921035 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.589820+0000", 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034765099999999998 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.589889+0000", 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8164999999999995e-05 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.589938+0000", 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9842999999999997e-05 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.969 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:49 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575826+0000", 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.376384114, 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018237082000000002, 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 49 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575826+0000", 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575826+0000", 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575827+0000", 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0699999999999996e-07 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575834+0000", 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8870000000000003e-06 2024-09-16T02:14:21.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575843+0000", 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6120000000000005e-06 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575853+0000", 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2159999999999995e-06 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.590021+0000", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014168627 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.590157+0000", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135413 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.590361+0000", 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000204518 2024-09-16T02:14:21.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.593773+0000", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034114129999999999 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.593811+0000", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8911000000000001e-05 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.593957+0000", 2024-09-16T02:14:21.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014515599999999999 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594063+0000", 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010652199999999999 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.376339087, 2024-09-16T02:14:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019551556000000001, 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 50 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575872+0000", 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5700000000000002e-07 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575960+0000", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8003000000000004e-05 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575971+0000", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1209000000000001e-05 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575979+0000", 2024-09-16T02:14:21.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8800000000000008e-06 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591098+0000", 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015119085000000001 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591336+0000", 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000238426 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591731+0000", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000394046 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594175+0000", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024441599999999999 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595149+0000", 2024-09-16T02:14:21.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097475000000000003 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595306+0000", 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000156918 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595423+0000", 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116322 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.376215949000001, 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019744136999999998, 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 51 2024-09-16T02:14:21.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:21.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575995+0000", 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3300000000000001e-07 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576002+0000", 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9190000000000002e-06 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576012+0000", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9690000000000006e-06 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576020+0000", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5839999999999997e-06 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.592718+0000", 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016698369000000001 2024-09-16T02:14:21.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.592850+0000", 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131877 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.593052+0000", 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020253799999999999 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595451+0000", 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023987660000000001 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595533+0000", 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2224000000000004e-05 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595617+0000", 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4136999999999998e-05 2024-09-16T02:14:21.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595738+0000", 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120921 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.376177865000001, 2024-09-16T02:14:21.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020204412000000001, 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576033+0000", 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6599999999999995e-07 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576055+0000", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1931999999999999e-05 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576066+0000", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1369000000000001e-05 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576074+0000", 2024-09-16T02:14:21.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5630000000000003e-06 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594265+0000", 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018191161000000001 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594479+0000", 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021409700000000001 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594685+0000", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000205708 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595917+0000", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232564 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596058+0000", 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014070900000000001 2024-09-16T02:14:21.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596135+0000", 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6878000000000005e-05 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596237+0000", 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101665 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:21.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:75 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.679740+0000", 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.272470621, 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014850619000000001, 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 75 2024-09-16T02:14:21.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679740+0000", 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679740+0000", 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:21.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679741+0000", 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1400000000000001e-06 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679876+0000", 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013517 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679899+0000", 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2648000000000001e-05 2024-09-16T02:14:21.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679915+0000", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6115000000000001e-05 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.687554+0000", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007639669 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.687679+0000", 2024-09-16T02:14:21.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012438099999999999 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.687880+0000", 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000201151 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.690454+0000", 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025739230000000001 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.694454+0000", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040003520000000004 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.694532+0000", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7464000000000002e-05 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.694590+0000", 2024-09-16T02:14:22.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8606000000000001e-05 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:76 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.679966+0000", 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.272244495000001, 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017233476000000001, 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.002 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 76 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679966+0000", 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679966+0000", 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679967+0000", 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4729999999999999e-06 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679978+0000", 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0723000000000001e-05 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.679996+0000", 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8108999999999999e-05 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.680032+0000", 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6103999999999999e-05 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.688877+0000", 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088451580000000005 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.689005+0000", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012813000000000001 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.689204+0000", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019855099999999999 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.690479+0000", 2024-09-16T02:14:22.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001274898 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.697041+0000", 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065623959999999999 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.697134+0000", 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2613000000000003e-05 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.697199+0000", 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5320999999999993e-05 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:95 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.796980+0000", 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.155230604, 2024-09-16T02:14:22.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018404182000000002, 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 95 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.796980+0000", 2024-09-16T02:14:22.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.796980+0000", 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.796981+0000", 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1960000000000001e-06 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.796991+0000", 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0019e-05 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797008+0000", 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7513e-05 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797022+0000", 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4165000000000001e-05 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.806581+0000", 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095583179999999997 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.806710+0000", 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000129107 2024-09-16T02:14:22.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.806903+0000", 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000193532 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811010+0000", 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041069540000000003 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815192+0000", 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041815990000000003 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815271+0000", 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8701000000000004e-05 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815384+0000", 2024-09-16T02:14:22.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113078 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797068+0000", 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.155142490999999, 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018641240999999999, 2024-09-16T02:14:22.016 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 96 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797068+0000", 2024-09-16T02:14:22.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797068+0000", 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797069+0000", 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797204+0000", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013510700000000001 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797225+0000", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0511999999999999e-05 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797240+0000", 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5065000000000001e-05 2024-09-16T02:14:22.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.807753+0000", 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010512850000000001 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.807870+0000", 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011718399999999999 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.808053+0000", 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018357900000000001 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813745+0000", 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056917449999999998 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815560+0000", 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001814604 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815650+0000", 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9776000000000002e-05 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815709+0000", 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9395999999999998e-05 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.022 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797266+0000", 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.154944102, 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019249157999999999, 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 97 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797266+0000", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797266+0000", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797268+0000", 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5689999999999999e-06 2024-09-16T02:14:22.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797279+0000", 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1260000000000001e-05 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797297+0000", 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8077999999999999e-05 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797311+0000", 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3602e-05 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.809564+0000", 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012253282000000001 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.809808+0000", 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024390499999999999 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.810151+0000", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000343634 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815414+0000", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052622210000000001 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.816389+0000", 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097577999999999999 2024-09-16T02:14:22.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.816460+0000", 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0823999999999998e-05 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.816515+0000", 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5003000000000002e-05 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.154871334999999, 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019987173, 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.029 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797340+0000", 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-09-16T02:14:22.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797353+0000", 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3156000000000001e-05 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797372+0000", 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8382000000000001e-05 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797386+0000", 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4262e-05 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811114+0000", 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013728557000000001 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811354+0000", 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023977499999999999 2024-09-16T02:14:22.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811632+0000", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027806899999999998 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815429+0000", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037971509999999999 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817182+0000", 2024-09-16T02:14:22.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001752382 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817262+0000", 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0240000000000004e-05 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817326+0000", 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4022000000000005e-05 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.154799853, 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020141841000000001, 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.035 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797412+0000", 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.243e-06 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797542+0000", 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077400000000001 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797562+0000", 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9644e-05 2024-09-16T02:14:22.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797577+0000", 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812322+0000", 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014744597999999999 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812456+0000", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134466 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812671+0000", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000214537 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815454+0000", 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027836179999999999 2024-09-16T02:14:22.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817415+0000", 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019603979999999999 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817490+0000", 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5012000000000006e-05 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817552+0000", 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2331999999999999e-05 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.154607148, 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020132852, 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.041 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:22.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797605+0000", 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7069999999999999e-06 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797620+0000", 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5585000000000001e-05 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797639+0000", 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8212000000000001e-05 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797658+0000", 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9514999999999999e-05 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813882+0000", 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016224045999999999 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814108+0000", 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022626600000000001 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814462+0000", 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035324400000000001 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815978+0000", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015162000000000001 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817614+0000", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001636366 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817684+0000", 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9701000000000002e-05 2024-09-16T02:14:22.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817736+0000", 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2009999999999998e-05 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:119 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.908728+0000", 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.043482631, 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016486872, 2024-09-16T02:14:22.047 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 119 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908728+0000", 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908728+0000", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908728+0000", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.47e-07 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908736+0000", 2024-09-16T02:14:22.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2130000000000002e-06 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908749+0000", 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3689000000000001e-05 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908765+0000", 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5125e-05 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.918296+0000", 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095311830000000004 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.918434+0000", 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000138276 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.918640+0000", 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020573100000000001 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922112+0000", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003472312 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925055+0000", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029426650000000001 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925146+0000", 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1199999999999994e-05 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925214+0000", 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8530999999999998e-05 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:120 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.908819+0000", 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.043391418000001, 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016864784000000001, 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.053 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 120 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908819+0000", 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908819+0000", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908820+0000", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908964+0000", 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014407600000000001 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908979+0000", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5322e-05 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.908989+0000", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3130000000000004e-06 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.919318+0000", 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010329603 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.919442+0000", 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012385099999999999 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.919634+0000", 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019147899999999999 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922119+0000", 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024852390000000002 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925376+0000", 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032568950000000001 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925530+0000", 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154568 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925684+0000", 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015323499999999999 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:121 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909031+0000", 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.043178877000001, 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018015450999999998, 2024-09-16T02:14:22.059 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 121 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909031+0000", 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909031+0000", 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909032+0000", 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0499999999999999e-06 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909039+0000", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0960000000000003e-06 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909054+0000", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4379000000000001e-05 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909062+0000", 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2190000000000004e-06 2024-09-16T02:14:22.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.920619+0000", 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01155743 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.920748+0000", 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128028 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.920944+0000", 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019629200000000001 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925235+0000", 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004291507 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926798+0000", 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001562595 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926947+0000", 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014896899999999999 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927047+0000", 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9885999999999999e-05 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.067 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.043132967, 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019924002999999999, 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 122 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909078+0000", 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.23e-07 2024-09-16T02:14:22.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909084+0000", 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000002e-06 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909094+0000", 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8330000000000006e-06 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909104+0000", 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0781999999999999e-05 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922447+0000", 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013342309 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922571+0000", 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012409699999999999 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922765+0000", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000193869 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926685+0000", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003919952 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928862+0000", 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002177216 2024-09-16T02:14:22.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928940+0000", 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8281000000000005e-05 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.929001+0000", 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1030999999999999e-05 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.043092127, 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:14:22.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:14:22.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:14:22.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:14:22.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 15.042967732999999, 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:22.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:14:22.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:14:22.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:22.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:22.086 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:22.088 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:14:22.088 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:22.087+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:22.189 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:22.184+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:22.219 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:22.219 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:22.219 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.234 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:14:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:14:22.247 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-16T02:14:22.248 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T02:14:22.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T02:14:22.290 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:22.287+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:22.364 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:22.364 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:22.364 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:14:22.390 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:22.386+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:22.491 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:22.491+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:22.508 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.508 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:22.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:22.508 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:14:22.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop(client.4293.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.174294+0000", 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.333828748, 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00193947, 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174294+0000", 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174294+0000", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174294+0000", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6700000000000003e-07 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174305+0000", 2024-09-16T02:14:22.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0926e-05 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174357+0000", 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1731000000000003e-05 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174371+0000", 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.418e-05 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174444+0000", 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2645000000000007e-05 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.174467+0000", 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2714e-05 2024-09-16T02:14:22.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.176184+0000", 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001717587 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.176233+0000", 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8919999999999999e-05 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.317590735, 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:14:22.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:14:22.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:14:22.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.857861+0000", 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.650261885999999, 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036258999999999998, 2024-09-16T02:14:22.520 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 3 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857861+0000", 2024-09-16T02:14:22.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857861+0000", 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857863+0000", 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2230000000000001e-06 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857873+0000", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0582e-05 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857900+0000", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6514999999999999e-05 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857916+0000", 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6586e-05 2024-09-16T02:14:22.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.858060+0000", 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014369900000000001 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.858223+0000", 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016298500000000001 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.524 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.342645315, 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037540080000000001, 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:22.525 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 5 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165479+0000", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.323e-06 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165493+0000", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3295999999999999e-05 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165525+0000", 2024-09-16T02:14:22.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2118e-05 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165542+0000", 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7176e-05 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165651+0000", 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010935200000000001 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165890+0000", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023896599999999999 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.166148+0000", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025728900000000001 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.167513+0000", 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013649999999999999 2024-09-16T02:14:22.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168951+0000", 2024-09-16T02:14:22.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014381719999999999 2024-09-16T02:14:22.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.169111+0000", 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000160175 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.169231+0000", 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120141 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4293.0:5 2.2 e26/25 ondisk, result = 0)", 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:09.168692+0000", 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.339430092000001, 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00055813800000000002, 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168692+0000", 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168692+0000", 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168694+0000", 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8199999999999999e-06 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168704+0000", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168773+0000", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8504999999999995e-05 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168787+0000", 2024-09-16T02:14:22.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3856000000000001e-05 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168906+0000", 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119425 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168928+0000", 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2172e-05 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.169251+0000", 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00032240399999999998 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.332564498, 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066455200000000002, 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.536 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175561+0000", 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7030000000000002e-06 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175575+0000", 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3889e-05 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175608+0000", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3704000000000001e-05 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175627+0000", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175763+0000", 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013625000000000001 2024-09-16T02:14:22.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175774+0000", 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.116e-05 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179478+0000", 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037039989999999999 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:22.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179657+0000", 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179538 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179882+0000", 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022455700000000001 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181213+0000", 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013309039999999999 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182065+0000", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085171900000000002 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182143+0000", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7904999999999994e-05 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182203+0000", 2024-09-16T02:14:22.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0909000000000002e-05 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_repop_reply(client.4293.0:7 2.2 e28/25 ondisk, result = 0)", 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:10.181852+0000", 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.326270225, 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036411599999999999, 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:22.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181852+0000", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181852+0000", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181854+0000", 2024-09-16T02:14:22.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9740000000000001e-06 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181864+0000", 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0165e-05 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181944+0000", 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9385e-05 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:22.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181962+0000", 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7776000000000001e-05 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182045+0000", 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3283999999999997e-05 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182057+0000", 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1839e-05 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:22.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182216+0000", 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159693 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:22.547 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.548 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:14:22.592 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:22.591+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:22.671 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.671 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:22.671 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:22.671 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.685 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:14:22.733 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:22.732+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:22.793 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:22.791+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:22.810 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.810 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:22.810 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:22.811 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:22.811 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.822 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:14:22.895 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:22.892+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:22.984 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:22.984 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [] 2024-09-16T02:14:22.984 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:22.996 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:22.992+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:22.998 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:14:23.096 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:23.095+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:23.120 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.120 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:23.120 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:23.120 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.134 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:14:23.198 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:23.195+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:23.250 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.250 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:23.250 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:23.251 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:23.251 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.257 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T02:14:23.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:14:23.281 DEBUG:teuthology.orchestra.run.smithi195:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0 2024-09-16T02:14:23.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T02:14:23.303 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight read -> 100 2024-09-16T02:14:23.303 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight delete -> 50 2024-09-16T02:14:23.303 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight snap_create -> 50 2024-09-16T02:14:23.303 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight snap_remove -> 50 2024-09-16T02:14:23.303 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight rollback -> 50 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight setattr -> 25 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight rmattr -> 25 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight copy_from -> 50 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:adding op weight append -> 100 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev) 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout:Configuration: 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout: Number of operations: 400 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout: Number of objects: 50 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout: Max in flight operations: 16 2024-09-16T02:14:23.304 INFO:tasks.rados.rados.0.smithi195.stdout: Object size (in bytes): 4000000 2024-09-16T02:14:23.305 INFO:tasks.rados.rados.0.smithi195.stdout: Write stride min: 400000 2024-09-16T02:14:23.305 INFO:tasks.rados.rados.0.smithi195.stdout: Write stride max: 800000 2024-09-16T02:14:23.309 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:23.308+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:23.383 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.383 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:23.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:23.383 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [] 2024-09-16T02:14:23.383 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.399 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:14:23.400 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:23.394+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:23.501 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:23.499+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:23.565 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.566 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:23.566 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:23.566 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.580 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:14:23.629 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:23.628+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:23.698 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.698 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:23.698 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:23.698 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:23.698 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.703 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:23.702+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:23.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:14:23.803 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:23.803+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:23.877 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:23.877 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:23.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:23.877 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [] 2024-09-16T02:14:23.877 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:23.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:23.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:14:23.904 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:23.903+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:24.005 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:23.998+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:24.024 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:24.024 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:24.024 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:24.039 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:14:24.047 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:24.047 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:24.074 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:24.074 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:24.102 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:24.106 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:24.098+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:24.132 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:24.132 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:24.161 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.161 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:24.162 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:24.162 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:24.175 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:14:24.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:24.189 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:24.207 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:24.204+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:24.220 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:24.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:24.265 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T02:14:24.270 INFO:tasks.rados.rados.0.smithi195.stdout:1: write initial oid 1 2024-09-16T02:14:24.270 INFO:tasks.rados.rados.0.smithi195.stdout:1: seq_num 0 ranges {0=32768} 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:1: writing smithi19530025-1 from 0 to 32768 tid 1 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:2: write initial oid 2 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:2: seq_num 1 ranges {0=32768} 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:2: writing smithi19530025-2 from 0 to 32768 tid 1 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:3: write initial oid 3 2024-09-16T02:14:24.271 INFO:tasks.rados.rados.0.smithi195.stdout:3: seq_num 2 ranges {0=49152} 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:3: writing smithi19530025-3 from 0 to 49152 tid 1 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:4: write initial oid 4 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:4: seq_num 3 ranges {0=65536} 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:4: writing smithi19530025-4 from 0 to 65536 tid 1 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:5: write initial oid 5 2024-09-16T02:14:24.272 INFO:tasks.rados.rados.0.smithi195.stdout:5: seq_num 4 ranges {0=81920} 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:5: writing smithi19530025-5 from 0 to 81920 tid 1 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:6: write initial oid 6 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:6: seq_num 5 ranges {0=98304} 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:6: writing smithi19530025-6 from 0 to 98304 tid 1 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:7: write initial oid 7 2024-09-16T02:14:24.273 INFO:tasks.rados.rados.0.smithi195.stdout:7: seq_num 6 ranges {0=114688} 2024-09-16T02:14:24.274 INFO:tasks.rados.rados.0.smithi195.stdout:7: writing smithi19530025-7 from 0 to 114688 tid 1 2024-09-16T02:14:24.274 INFO:tasks.rados.rados.0.smithi195.stdout:8: write initial oid 8 2024-09-16T02:14:24.274 INFO:tasks.rados.rados.0.smithi195.stdout:8: seq_num 7 ranges {0=131072} 2024-09-16T02:14:24.275 INFO:tasks.rados.rados.0.smithi195.stdout:8: writing smithi19530025-8 from 0 to 131072 tid 1 2024-09-16T02:14:24.275 INFO:tasks.rados.rados.0.smithi195.stdout:9: write initial oid 9 2024-09-16T02:14:24.275 INFO:tasks.rados.rados.0.smithi195.stdout:9: seq_num 8 ranges {0=147456} 2024-09-16T02:14:24.276 INFO:tasks.rados.rados.0.smithi195.stdout:9: writing smithi19530025-9 from 0 to 147456 tid 1 2024-09-16T02:14:24.277 INFO:tasks.rados.rados.0.smithi195.stdout:10: write initial oid 10 2024-09-16T02:14:24.277 INFO:tasks.rados.rados.0.smithi195.stdout:10: seq_num 9 ranges {0=163840} 2024-09-16T02:14:24.278 INFO:tasks.rados.rados.0.smithi195.stdout:10: writing smithi19530025-10 from 0 to 163840 tid 1 2024-09-16T02:14:24.278 INFO:tasks.rados.rados.0.smithi195.stdout:11: write initial oid 11 2024-09-16T02:14:24.278 INFO:tasks.rados.rados.0.smithi195.stdout:11: seq_num 10 ranges {0=180224} 2024-09-16T02:14:24.280 INFO:tasks.rados.rados.0.smithi195.stdout:11: writing smithi19530025-11 from 0 to 180224 tid 1 2024-09-16T02:14:24.280 INFO:tasks.rados.rados.0.smithi195.stdout:12: write initial oid 12 2024-09-16T02:14:24.281 INFO:tasks.rados.rados.0.smithi195.stdout:12: seq_num 11 ranges {0=196608} 2024-09-16T02:14:24.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:14:24.283 INFO:tasks.rados.rados.0.smithi195.stdout:12: writing smithi19530025-12 from 0 to 196608 tid 1 2024-09-16T02:14:24.285 INFO:tasks.rados.rados.0.smithi195.stdout:13: write initial oid 13 2024-09-16T02:14:24.285 INFO:tasks.rados.rados.0.smithi195.stdout:13: seq_num 12 ranges {0=212992} 2024-09-16T02:14:24.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:24.285 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:24.286 INFO:tasks.rados.rados.0.smithi195.stdout:13: writing smithi19530025-13 from 0 to 212992 tid 1 2024-09-16T02:14:24.286 INFO:tasks.rados.rados.0.smithi195.stdout:14: write initial oid 14 2024-09-16T02:14:24.286 INFO:tasks.rados.rados.0.smithi195.stdout:14: seq_num 13 ranges {0=229376} 2024-09-16T02:14:24.288 INFO:tasks.rados.rados.0.smithi195.stdout:14: writing smithi19530025-14 from 0 to 229376 tid 1 2024-09-16T02:14:24.288 INFO:tasks.rados.rados.0.smithi195.stdout:15: write initial oid 15 2024-09-16T02:14:24.288 INFO:tasks.rados.rados.0.smithi195.stdout:15: seq_num 14 ranges {0=245760} 2024-09-16T02:14:24.289 INFO:tasks.rados.rados.0.smithi195.stdout:15: writing smithi19530025-15 from 0 to 245760 tid 1 2024-09-16T02:14:24.289 INFO:tasks.rados.rados.0.smithi195.stdout:16: write initial oid 16 2024-09-16T02:14:24.289 INFO:tasks.rados.rados.0.smithi195.stdout:16: seq_num 15 ranges {0=262144} 2024-09-16T02:14:24.291 INFO:tasks.rados.rados.0.smithi195.stdout:16: writing smithi19530025-16 from 0 to 262144 tid 1 2024-09-16T02:14:24.291 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.291 INFO:tasks.rados.rados.0.smithi195.stdout:4: finishing write tid 1 to smithi19530025-4 2024-09-16T02:14:24.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T02:14:24.294 INFO:tasks.rados.rados.0.smithi195.stdout:3: finishing write tid 1 to smithi19530025-3 2024-09-16T02:14:24.294 INFO:tasks.rados.rados.0.smithi195.stdout:7: finishing write tid 1 to smithi19530025-7 2024-09-16T02:14:24.296 INFO:tasks.rados.rados.0.smithi195.stdout:6: finishing write tid 1 to smithi19530025-6 2024-09-16T02:14:24.296 INFO:tasks.rados.rados.0.smithi195.stdout:6: finishing write tid 2 to smithi19530025-6 2024-09-16T02:14:24.299 INFO:tasks.rados.rados.0.smithi195.stdout:3: finishing write tid 2 to smithi19530025-3 2024-09-16T02:14:24.300 INFO:tasks.rados.rados.0.smithi195.stdout:2: finishing write tid 1 to smithi19530025-2 2024-09-16T02:14:24.300 INFO:tasks.rados.rados.0.smithi195.stdout:2: finishing write tid 2 to smithi19530025-2 2024-09-16T02:14:24.301 INFO:tasks.rados.rados.0.smithi195.stdout:4: finishing write tid 2 to smithi19530025-4 2024-09-16T02:14:24.302 INFO:tasks.rados.rados.0.smithi195.stdout:1: finishing write tid 1 to smithi19530025-1 2024-09-16T02:14:24.302 INFO:tasks.rados.rados.0.smithi195.stdout:1: finishing write tid 2 to smithi19530025-1 2024-09-16T02:14:24.302 INFO:tasks.rados.rados.0.smithi195.stdout:3: finishing write tid 3 to smithi19530025-3 2024-09-16T02:14:24.302 INFO:tasks.rados.rados.0.smithi195.stdout:3: oid 3 updating version 0 to 1 2024-09-16T02:14:24.302 INFO:tasks.rados.rados.0.smithi195.stdout:3: oid 3 updating version 1 to 2 2024-09-16T02:14:24.303 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 3 v 2 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-16T02:14:24.303 INFO:tasks.rados.rados.0.smithi195.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-16T02:14:24.303 INFO:tasks.rados.rados.0.smithi195.stdout:3: done (15 left) 2024-09-16T02:14:24.303 INFO:tasks.rados.rados.0.smithi195.stdout:17: write initial oid 17 2024-09-16T02:14:24.303 INFO:tasks.rados.rados.0.smithi195.stdout:17: seq_num 16 ranges {0=278528} 2024-09-16T02:14:24.305 INFO:tasks.rados.rados.0.smithi195.stdout:17: writing smithi19530025-17 from 0 to 278528 tid 1 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: finishing write tid 2 to smithi19530025-7 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:6: finishing write tid 3 to smithi19530025-6 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:6: oid 6 updating version 0 to 1 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:6: oid 6 updating version 1 to 2 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 6 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: finishing write tid 3 to smithi19530025-7 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: oid 7 updating version 0 to 1 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: oid 7 updating version 1 to 2 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 2 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:6: done (15 left) 2024-09-16T02:14:24.306 INFO:tasks.rados.rados.0.smithi195.stdout:7: done (14 left) 2024-09-16T02:14:24.307 INFO:tasks.rados.rados.0.smithi195.stdout:18: write initial oid 18 2024-09-16T02:14:24.307 INFO:tasks.rados.rados.0.smithi195.stdout:18: seq_num 17 ranges {0=294912} 2024-09-16T02:14:24.307 INFO:tasks.rados.rados.0.smithi195.stdout:18: writing smithi19530025-18 from 0 to 294912 tid 1 2024-09-16T02:14:24.307 INFO:tasks.rados.rados.0.smithi195.stdout:19: write initial oid 19 2024-09-16T02:14:24.308 INFO:tasks.rados.rados.0.smithi195.stdout:19: seq_num 18 ranges {0=311296} 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:19: writing smithi19530025-19 from 0 to 311296 tid 1 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:10: finishing write tid 1 to smithi19530025-10 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:1: finishing write tid 3 to smithi19530025-1 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:1: oid 1 updating version 0 to 1 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:1: oid 1 updating version 1 to 2 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 1 v 2 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:15: finishing write tid 1 to smithi19530025-15 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:15: finishing write tid 2 to smithi19530025-15 2024-09-16T02:14:24.310 INFO:tasks.rados.rados.0.smithi195.stdout:10: finishing write tid 2 to smithi19530025-10 2024-09-16T02:14:24.311 INFO:tasks.rados.rados.0.smithi195.stdout:1: done (15 left) 2024-09-16T02:14:24.311 INFO:tasks.rados.rados.0.smithi195.stdout:20: write initial oid 20 2024-09-16T02:14:24.311 INFO:tasks.rados.rados.0.smithi195.stdout:20: seq_num 19 ranges {0=327680} 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:20: writing smithi19530025-20 from 0 to 327680 tid 1 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:2: finishing write tid 3 to smithi19530025-2 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:2: oid 2 updating version 0 to 2 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:2: oid 2 version 2 is already newer than 1 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 2 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-16T02:14:24.312 INFO:tasks.rados.rados.0.smithi195.stdout:2: done (15 left) 2024-09-16T02:14:24.313 INFO:tasks.rados.rados.0.smithi195.stdout:21: write initial oid 21 2024-09-16T02:14:24.313 INFO:tasks.rados.rados.0.smithi195.stdout:21: seq_num 20 ranges {0=344064} 2024-09-16T02:14:24.314 INFO:tasks.rados.rados.0.smithi195.stdout:21: writing smithi19530025-21 from 0 to 344064 tid 1 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:15: finishing write tid 3 to smithi19530025-15 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:15: oid 15 updating version 0 to 3 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:15: oid 15 updating version 3 to 4 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 15 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:11: finishing write tid 1 to smithi19530025-11 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:15: done (15 left) 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:22: write initial oid 22 2024-09-16T02:14:24.315 INFO:tasks.rados.rados.0.smithi195.stdout:22: seq_num 21 ranges {0=360448} 2024-09-16T02:14:24.317 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.317 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:22: writing smithi19530025-22 from 0 to 360448 tid 1 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:11: finishing write tid 2 to smithi19530025-11 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:13: finishing write tid 1 to smithi19530025-13 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:13: finishing write tid 2 to smithi19530025-13 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:9: finishing write tid 1 to smithi19530025-9 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:16: finishing write tid 1 to smithi19530025-16 2024-09-16T02:14:24.318 INFO:tasks.rados.rados.0.smithi195.stdout:16: finishing write tid 2 to smithi19530025-16 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:11: finishing write tid 3 to smithi19530025-11 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:11: oid 11 updating version 0 to 1 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:11: oid 11 updating version 1 to 2 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 2 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:13: finishing write tid 3 to smithi19530025-13 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:13: oid 13 updating version 0 to 3 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:13: oid 13 updating version 3 to 4 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 13 v 4 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:14: finishing write tid 1 to smithi19530025-14 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:14: finishing write tid 2 to smithi19530025-14 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:12: finishing write tid 1 to smithi19530025-12 2024-09-16T02:14:24.319 INFO:tasks.rados.rados.0.smithi195.stdout:11: done (15 left) 2024-09-16T02:14:24.320 INFO:tasks.rados.rados.0.smithi195.stdout:13: done (14 left) 2024-09-16T02:14:24.320 INFO:tasks.rados.rados.0.smithi195.stdout:23: write initial oid 23 2024-09-16T02:14:24.320 INFO:tasks.rados.rados.0.smithi195.stdout:23: seq_num 22 ranges {0=376832} 2024-09-16T02:14:24.320 INFO:tasks.rados.rados.0.smithi195.stdout:23: writing smithi19530025-23 from 0 to 376832 tid 1 2024-09-16T02:14:24.320 INFO:tasks.rados.rados.0.smithi195.stdout:24: write initial oid 24 2024-09-16T02:14:24.321 INFO:tasks.rados.rados.0.smithi195.stdout:5: finishing write tid 1 to smithi19530025-5 2024-09-16T02:14:24.321 INFO:tasks.rados.rados.0.smithi195.stdout:12: finishing write tid 2 to smithi19530025-12 2024-09-16T02:14:24.321 INFO:tasks.rados.rados.0.smithi195.stdout:18: finishing write tid 1 to smithi19530025-18 2024-09-16T02:14:24.321 INFO:tasks.rados.rados.0.smithi195.stdout:24: seq_num 23 ranges {0=393216} 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:24: writing smithi19530025-24 from 0 to 393216 tid 1 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:9: finishing write tid 2 to smithi19530025-9 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:10: finishing write tid 3 to smithi19530025-10 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:10: oid 10 updating version 0 to 2 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:10: oid 10 version 2 is already newer than 1 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 10 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:14: finishing write tid 3 to smithi19530025-14 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:14: oid 14 updating version 0 to 5 2024-09-16T02:14:24.324 INFO:tasks.rados.rados.0.smithi195.stdout:14: oid 14 updating version 5 to 6 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:16: finishing write tid 3 to smithi19530025-16 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:16: oid 16 updating version 0 to 2 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:16: oid 16 version 2 is already newer than 1 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 16 v 2 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:12: finishing write tid 3 to smithi19530025-12 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:12: oid 12 updating version 0 to 1 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:12: oid 12 updating version 1 to 2 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 12 v 2 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-16T02:14:24.325 INFO:tasks.rados.rados.0.smithi195.stdout:10: done (15 left) 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:12: done (14 left) 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:14: done (13 left) 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:16: done (12 left) 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:25: write initial oid 25 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:25: seq_num 24 ranges {0=409600} 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:25: writing smithi19530025-25 from 0 to 409600 tid 1 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:26: write initial oid 26 2024-09-16T02:14:24.326 INFO:tasks.rados.rados.0.smithi195.stdout:26: seq_num 25 ranges {0=425984} 2024-09-16T02:14:24.328 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:24.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:24.329 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:24.322+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:24.329 INFO:tasks.rados.rados.0.smithi195.stdout:26: writing smithi19530025-26 from 0 to 425984 tid 1 2024-09-16T02:14:24.330 INFO:tasks.rados.rados.0.smithi195.stdout:27: write initial oid 27 2024-09-16T02:14:24.330 INFO:tasks.rados.rados.0.smithi195.stdout:27: seq_num 26 ranges {0=442368} 2024-09-16T02:14:24.333 INFO:tasks.rados.rados.0.smithi195.stdout:27: writing smithi19530025-27 from 0 to 442368 tid 1 2024-09-16T02:14:24.333 INFO:tasks.rados.rados.0.smithi195.stdout:28: write initial oid 28 2024-09-16T02:14:24.333 INFO:tasks.rados.rados.0.smithi195.stdout:28: seq_num 27 ranges {0=458752} 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:28: writing smithi19530025-28 from 0 to 458752 tid 1 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:21: finishing write tid 1 to smithi19530025-21 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:5: finishing write tid 2 to smithi19530025-5 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:4: finishing write tid 3 to smithi19530025-4 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:4: oid 4 updating version 0 to 1 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:4: oid 4 updating version 1 to 2 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 4 v 2 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-16T02:14:24.337 INFO:tasks.rados.rados.0.smithi195.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:19: finishing write tid 1 to smithi19530025-19 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:21: finishing write tid 2 to smithi19530025-21 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:19: finishing write tid 2 to smithi19530025-19 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:17: finishing write tid 1 to smithi19530025-17 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:17: finishing write tid 2 to smithi19530025-17 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:22: finishing write tid 1 to smithi19530025-22 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:22: finishing write tid 2 to smithi19530025-22 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:21: finishing write tid 3 to smithi19530025-21 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:21: oid 21 updating version 0 to 4 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:21: oid 21 version 4 is already newer than 3 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 21 v 4 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:14:24.338 INFO:tasks.rados.rados.0.smithi195.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:19: finishing write tid 3 to smithi19530025-19 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:19: oid 19 updating version 0 to 4 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:19: oid 19 version 4 is already newer than 3 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 19 v 4 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:22: finishing write tid 3 to smithi19530025-22 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:22: oid 22 updating version 0 to 7 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:22: oid 22 updating version 7 to 8 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 22 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:23: finishing write tid 1 to smithi19530025-23 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:20: finishing write tid 1 to smithi19530025-20 2024-09-16T02:14:24.339 INFO:tasks.rados.rados.0.smithi195.stdout:17: finishing write tid 3 to smithi19530025-17 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:17: oid 17 updating version 0 to 2 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:17: oid 17 version 2 is already newer than 1 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 17 v 2 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:5: finishing write tid 3 to smithi19530025-5 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:5: oid 5 updating version 0 to 3 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:5: oid 5 updating version 3 to 4 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 5 v 4 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:4: done (15 left) 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:5: done (14 left) 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:17: done (13 left) 2024-09-16T02:14:24.340 INFO:tasks.rados.rados.0.smithi195.stdout:19: done (12 left) 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:21: done (11 left) 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:22: done (10 left) 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:29: write initial oid 29 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:29: seq_num 28 ranges {0=475136} 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:29: writing smithi19530025-29 from 0 to 475136 tid 1 2024-09-16T02:14:24.341 INFO:tasks.rados.rados.0.smithi195.stdout:30: write initial oid 30 2024-09-16T02:14:24.342 INFO:tasks.rados.rados.0.smithi195.stdout:30: seq_num 29 ranges {0=491520} 2024-09-16T02:14:24.343 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:14:24.344 INFO:tasks.rados.rados.0.smithi195.stdout:30: writing smithi19530025-30 from 0 to 491520 tid 1 2024-09-16T02:14:24.344 INFO:tasks.rados.rados.0.smithi195.stdout:31: write initial oid 31 2024-09-16T02:14:24.344 INFO:tasks.rados.rados.0.smithi195.stdout:31: seq_num 30 ranges {0=507904} 2024-09-16T02:14:24.347 INFO:tasks.rados.rados.0.smithi195.stdout:31: writing smithi19530025-31 from 0 to 507904 tid 1 2024-09-16T02:14:24.347 INFO:tasks.rados.rados.0.smithi195.stdout:32: write initial oid 32 2024-09-16T02:14:24.347 INFO:tasks.rados.rados.0.smithi195.stdout:32: seq_num 31 ranges {0=524288} 2024-09-16T02:14:24.351 INFO:tasks.rados.rados.0.smithi195.stdout:32: writing smithi19530025-32 from 0 to 524288 tid 1 2024-09-16T02:14:24.351 INFO:tasks.rados.rados.0.smithi195.stdout:33: write initial oid 33 2024-09-16T02:14:24.351 INFO:tasks.rados.rados.0.smithi195.stdout:33: seq_num 32 ranges {0=540672} 2024-09-16T02:14:24.355 INFO:tasks.rados.rados.0.smithi195.stdout:33: writing smithi19530025-33 from 0 to 540672 tid 1 2024-09-16T02:14:24.355 INFO:tasks.rados.rados.0.smithi195.stdout:34: write initial oid 34 2024-09-16T02:14:24.355 INFO:tasks.rados.rados.0.smithi195.stdout:34: seq_num 33 ranges {0=557056} 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:34: writing smithi19530025-34 from 0 to 557056 tid 1 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:20: finishing write tid 2 to smithi19530025-20 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:20: finishing write tid 3 to smithi19530025-20 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:20: oid 20 updating version 0 to 6 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:20: oid 20 version 6 is already newer than 5 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-16T02:14:24.359 INFO:tasks.rados.rados.0.smithi195.stdout:8: finishing write tid 1 to smithi19530025-8 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: finishing write tid 2 to smithi19530025-8 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: finishing write tid 3 to smithi19530025-8 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: oid 8 updating version 0 to 5 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: oid 8 updating version 5 to 6 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:28: finishing write tid 1 to smithi19530025-28 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:28: finishing write tid 2 to smithi19530025-28 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:25: finishing write tid 1 to smithi19530025-25 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:29: finishing write tid 1 to smithi19530025-29 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:8: done (15 left) 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:20: done (14 left) 2024-09-16T02:14:24.360 INFO:tasks.rados.rados.0.smithi195.stdout:35: write initial oid 35 2024-09-16T02:14:24.361 INFO:tasks.rados.rados.0.smithi195.stdout:35: seq_num 34 ranges {0=573440} 2024-09-16T02:14:24.362 INFO:tasks.rados.rados.0.smithi195.stdout:35: writing smithi19530025-35 from 0 to 573440 tid 1 2024-09-16T02:14:24.362 INFO:tasks.rados.rados.0.smithi195.stdout:36: write initial oid 36 2024-09-16T02:14:24.362 INFO:tasks.rados.rados.0.smithi195.stdout:36: seq_num 35 ranges {0=589824} 2024-09-16T02:14:24.366 INFO:tasks.rados.rados.0.smithi195.stdout:36: writing smithi19530025-36 from 0 to 589824 tid 1 2024-09-16T02:14:24.366 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.366 INFO:tasks.rados.rados.0.smithi195.stdout:29: finishing write tid 2 to smithi19530025-29 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:24: finishing write tid 1 to smithi19530025-24 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:27: finishing write tid 1 to smithi19530025-27 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:28: finishing write tid 3 to smithi19530025-28 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:28: oid 28 updating version 0 to 3 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:28: oid 28 updating version 3 to 4 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 28 v 4 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:24: finishing write tid 2 to smithi19530025-24 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:27: finishing write tid 2 to smithi19530025-27 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:29: finishing write tid 3 to smithi19530025-29 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:29: oid 29 updating version 0 to 5 2024-09-16T02:14:24.367 INFO:tasks.rados.rados.0.smithi195.stdout:29: oid 29 updating version 5 to 6 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 29 v 6 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:30: finishing write tid 1 to smithi19530025-30 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:28: done (15 left) 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:29: done (14 left) 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:37: write initial oid 37 2024-09-16T02:14:24.368 INFO:tasks.rados.rados.0.smithi195.stdout:37: seq_num 36 ranges {0=606208} 2024-09-16T02:14:24.370 INFO:tasks.rados.rados.0.smithi195.stdout:37: writing smithi19530025-37 from 0 to 606208 tid 1 2024-09-16T02:14:24.370 INFO:tasks.rados.rados.0.smithi195.stdout:38: write initial oid 38 2024-09-16T02:14:24.370 INFO:tasks.rados.rados.0.smithi195.stdout:38: seq_num 37 ranges {0=622592} 2024-09-16T02:14:24.374 INFO:tasks.rados.rados.0.smithi195.stdout:38: writing smithi19530025-38 from 0 to 622592 tid 1 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:31: finishing write tid 1 to smithi19530025-31 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:27: finishing write tid 3 to smithi19530025-27 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:27: oid 27 updating version 0 to 3 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:27: oid 27 updating version 3 to 4 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 27 v 4 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:24: finishing write tid 3 to smithi19530025-24 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:24: oid 24 updating version 0 to 4 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:24: oid 24 version 4 is already newer than 3 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 24 v 4 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-16T02:14:24.375 INFO:tasks.rados.rados.0.smithi195.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-16T02:14:24.376 INFO:tasks.rados.rados.0.smithi195.stdout:23: finishing write tid 2 to smithi19530025-23 2024-09-16T02:14:24.376 INFO:tasks.rados.rados.0.smithi195.stdout:24: done (15 left) 2024-09-16T02:14:24.376 INFO:tasks.rados.rados.0.smithi195.stdout:27: done (14 left) 2024-09-16T02:14:24.376 INFO:tasks.rados.rados.0.smithi195.stdout:39: write initial oid 39 2024-09-16T02:14:24.376 INFO:tasks.rados.rados.0.smithi195.stdout:39: seq_num 38 ranges {0=638976} 2024-09-16T02:14:24.376 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:24.376 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:24.378 INFO:tasks.rados.rados.0.smithi195.stdout:39: writing smithi19530025-39 from 0 to 638976 tid 1 2024-09-16T02:14:24.378 INFO:tasks.rados.rados.0.smithi195.stdout:40: write initial oid 40 2024-09-16T02:14:24.378 INFO:tasks.rados.rados.0.smithi195.stdout:40: seq_num 39 ranges {0=671744} 2024-09-16T02:14:24.382 INFO:tasks.rados.rados.0.smithi195.stdout:40: writing smithi19530025-40 from 0 to 671744 tid 1 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:30: finishing write tid 2 to smithi19530025-30 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:26: finishing write tid 1 to smithi19530025-26 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:23: finishing write tid 3 to smithi19530025-23 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:23: oid 23 updating version 0 to 1 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:23: oid 23 updating version 1 to 2 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 23 v 2 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:32: finishing write tid 1 to smithi19530025-32 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:30: finishing write tid 3 to smithi19530025-30 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:30: oid 30 updating version 0 to 8 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:30: oid 30 version 8 is already newer than 7 2024-09-16T02:14:24.383 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 8 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-16T02:14:24.384 INFO:tasks.rados.rados.0.smithi195.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-16T02:14:24.384 INFO:tasks.rados.rados.0.smithi195.stdout:23: done (15 left) 2024-09-16T02:14:24.384 INFO:tasks.rados.rados.0.smithi195.stdout:30: done (14 left) 2024-09-16T02:14:24.384 INFO:tasks.rados.rados.0.smithi195.stdout:41: write initial oid 41 2024-09-16T02:14:24.384 INFO:tasks.rados.rados.0.smithi195.stdout:41: seq_num 40 ranges {0=688128} 2024-09-16T02:14:24.387 INFO:tasks.rados.rados.0.smithi195.stdout:41: writing smithi19530025-41 from 0 to 688128 tid 1 2024-09-16T02:14:24.387 INFO:tasks.rados.rados.0.smithi195.stdout:42: write initial oid 42 2024-09-16T02:14:24.387 INFO:tasks.rados.rados.0.smithi195.stdout:42: seq_num 41 ranges {0=704512} 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout:42: writing smithi19530025-42 from 0 to 704512 tid 1 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout:32: finishing write tid 2 to smithi19530025-32 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout:34: finishing write tid 1 to smithi19530025-34 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout:32: finishing write tid 3 to smithi19530025-32 2024-09-16T02:14:24.392 INFO:tasks.rados.rados.0.smithi195.stdout:32: oid 32 updating version 0 to 4 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:32: oid 32 version 4 is already newer than 3 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 32 v 4 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:34: finishing write tid 2 to smithi19530025-34 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:32: done (15 left) 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:43: write initial oid 43 2024-09-16T02:14:24.393 INFO:tasks.rados.rados.0.smithi195.stdout:43: seq_num 42 ranges {0=720896} 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:43: writing smithi19530025-43 from 0 to 720896 tid 1 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 16 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:33: finishing write tid 1 to smithi19530025-33 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:18: finishing write tid 2 to smithi19530025-18 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:33: finishing write tid 2 to smithi19530025-33 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:35: finishing write tid 1 to smithi19530025-35 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:18: finishing write tid 3 to smithi19530025-18 2024-09-16T02:14:24.398 INFO:tasks.rados.rados.0.smithi195.stdout:18: oid 18 updating version 0 to 3 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:18: oid 18 updating version 3 to 4 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 18 v 4 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:34: finishing write tid 3 to smithi19530025-34 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:34: oid 34 updating version 0 to 10 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:34: oid 34 version 10 is already newer than 9 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 10 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:36: finishing write tid 1 to smithi19530025-36 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:9: finishing write tid 3 to smithi19530025-9 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:9: oid 9 updating version 0 to 2 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:9: oid 9 version 2 is already newer than 1 2024-09-16T02:14:24.399 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 9 v 2 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:37: finishing write tid 1 to smithi19530025-37 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:36: finishing write tid 2 to smithi19530025-36 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:37: finishing write tid 2 to smithi19530025-37 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:36: finishing write tid 3 to smithi19530025-36 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:36: oid 36 updating version 0 to 11 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:36: oid 36 updating version 11 to 12 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:38: finishing write tid 1 to smithi19530025-38 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:33: finishing write tid 3 to smithi19530025-33 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:33: oid 33 updating version 0 to 2 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:33: oid 33 version 2 is already newer than 1 2024-09-16T02:14:24.400 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 33 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:38: finishing write tid 2 to smithi19530025-38 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:39: finishing write tid 1 to smithi19530025-39 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:38: finishing write tid 3 to smithi19530025-38 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:38: oid 38 updating version 0 to 14 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:38: oid 38 version 14 is already newer than 13 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 38 v 14 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:40: finishing write tid 1 to smithi19530025-40 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:40: finishing write tid 2 to smithi19530025-40 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:40: finishing write tid 3 to smithi19530025-40 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:40: oid 40 updating version 0 to 6 2024-09-16T02:14:24.401 INFO:tasks.rados.rados.0.smithi195.stdout:40: oid 40 version 6 is already newer than 5 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 40 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:41: finishing write tid 1 to smithi19530025-41 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:35: finishing write tid 2 to smithi19530025-35 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:35: finishing write tid 3 to smithi19530025-35 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:35: oid 35 updating version 0 to 5 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:35: oid 35 updating version 5 to 6 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:41: finishing write tid 2 to smithi19530025-41 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:42: finishing write tid 1 to smithi19530025-42 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:9: done (15 left) 2024-09-16T02:14:24.402 INFO:tasks.rados.rados.0.smithi195.stdout:18: done (14 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:33: done (13 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:34: done (12 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:35: done (11 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:36: done (10 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:38: done (9 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:40: done (8 left) 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:44: write initial oid 44 2024-09-16T02:14:24.403 INFO:tasks.rados.rados.0.smithi195.stdout:44: seq_num 43 ranges {0=737280} 2024-09-16T02:14:24.404 INFO:tasks.rados.rados.0.smithi195.stdout:44: writing smithi19530025-44 from 0 to 737280 tid 1 2024-09-16T02:14:24.404 INFO:tasks.rados.rados.0.smithi195.stdout:45: write initial oid 45 2024-09-16T02:14:24.404 INFO:tasks.rados.rados.0.smithi195.stdout:45: seq_num 44 ranges {0=753664} 2024-09-16T02:14:24.407 INFO:tasks.rados.rados.0.smithi195.stdout:45: writing smithi19530025-45 from 0 to 753664 tid 1 2024-09-16T02:14:24.407 INFO:tasks.rados.rados.0.smithi195.stdout:46: write initial oid 46 2024-09-16T02:14:24.407 INFO:tasks.rados.rados.0.smithi195.stdout:46: seq_num 45 ranges {0=770048} 2024-09-16T02:14:24.408 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:24.404+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:24.411 INFO:tasks.rados.rados.0.smithi195.stdout:46: writing smithi19530025-46 from 0 to 770048 tid 1 2024-09-16T02:14:24.412 INFO:tasks.rados.rados.0.smithi195.stdout:47: write initial oid 47 2024-09-16T02:14:24.412 INFO:tasks.rados.rados.0.smithi195.stdout:47: seq_num 46 ranges {0=786432} 2024-09-16T02:14:24.415 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:24.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:24.416 INFO:tasks.rados.rados.0.smithi195.stdout:47: writing smithi19530025-47 from 0 to 786432 tid 1 2024-09-16T02:14:24.416 INFO:tasks.rados.rados.0.smithi195.stdout:48: write initial oid 48 2024-09-16T02:14:24.416 INFO:tasks.rados.rados.0.smithi195.stdout:48: seq_num 47 ranges {0=425984,425984=376832} 2024-09-16T02:14:24.418 INFO:tasks.rados.rados.0.smithi195.stdout:48: writing smithi19530025-48 from 0 to 425984 tid 1 2024-09-16T02:14:24.421 INFO:tasks.rados.rados.0.smithi195.stdout:48: writing smithi19530025-48 from 425984 to 802816 tid 2 2024-09-16T02:14:24.421 INFO:tasks.rados.rados.0.smithi195.stdout:49: write initial oid 49 2024-09-16T02:14:24.421 INFO:tasks.rados.rados.0.smithi195.stdout:49: seq_num 48 ranges {0=442368,442368=376832} 2024-09-16T02:14:24.424 INFO:tasks.rados.rados.0.smithi195.stdout:49: writing smithi19530025-49 from 0 to 442368 tid 1 2024-09-16T02:14:24.427 INFO:tasks.rados.rados.0.smithi195.stdout:49: writing smithi19530025-49 from 442368 to 819200 tid 2 2024-09-16T02:14:24.427 INFO:tasks.rados.rados.0.smithi195.stdout:50: write initial oid 50 2024-09-16T02:14:24.427 INFO:tasks.rados.rados.0.smithi195.stdout:50: seq_num 49 ranges {0=458752,458752=376832} 2024-09-16T02:14:24.430 INFO:tasks.rados.rados.0.smithi195.stdout:50: writing smithi19530025-50 from 0 to 458752 tid 1 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:50: writing smithi19530025-50 from 458752 to 835584 tid 2 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:51: rollback oid 22 current snap is 0 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:51: done (15 left) 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:52: delete oid 35 current snap is 0 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:41: finishing write tid 3 to smithi19530025-41 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:41: oid 41 updating version 0 to 9 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:41: oid 41 updating version 9 to 10 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 41 v 10 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:42: finishing write tid 2 to smithi19530025-42 2024-09-16T02:14:24.434 INFO:tasks.rados.rados.0.smithi195.stdout:43: finishing write tid 1 to smithi19530025-43 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:42: finishing write tid 3 to smithi19530025-42 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:42: oid 42 updating version 0 to 8 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:42: oid 42 version 8 is already newer than 7 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 42 v 8 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:44: finishing write tid 1 to smithi19530025-44 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: finishing write tid 1 to smithi19530025-45 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: finishing write tid 2 to smithi19530025-45 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: finishing write tid 3 to smithi19530025-45 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: oid 45 updating version 0 to 7 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: oid 45 updating version 7 to 8 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 8 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-16T02:14:24.435 INFO:tasks.rados.rados.0.smithi195.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:37: finishing write tid 3 to smithi19530025-37 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:37: oid 37 updating version 0 to 4 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:37: oid 37 version 4 is already newer than 3 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:46: finishing write tid 1 to smithi19530025-46 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:46: finishing write tid 2 to smithi19530025-46 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:47: finishing write tid 1 to smithi19530025-47 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:47: finishing write tid 2 to smithi19530025-47 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:44: finishing write tid 2 to smithi19530025-44 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:47: finishing write tid 3 to smithi19530025-47 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:47: oid 47 updating version 0 to 2 2024-09-16T02:14:24.436 INFO:tasks.rados.rados.0.smithi195.stdout:47: oid 47 version 2 is already newer than 1 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 47 v 2 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:44: finishing write tid 3 to smithi19530025-44 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:44: oid 44 updating version 0 to 11 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:44: oid 44 updating version 11 to 12 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:48: finishing write tid 1 to smithi19530025-48 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:48: finishing write tid 2 to smithi19530025-48 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:43: finishing write tid 2 to smithi19530025-43 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:43: finishing write tid 3 to smithi19530025-43 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:43: oid 43 updating version 0 to 9 2024-09-16T02:14:24.437 INFO:tasks.rados.rados.0.smithi195.stdout:43: oid 43 updating version 9 to 10 2024-09-16T02:14:24.438 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 43 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T02:14:24.438 INFO:tasks.rados.rados.0.smithi195.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-16T02:14:24.438 INFO:tasks.rados.rados.0.smithi195.stdout:49: finishing write tid 1 to smithi19530025-49 2024-09-16T02:14:24.438 INFO:tasks.rados.rados.0.smithi195.stdout:49: finishing write tid 2 to smithi19530025-49 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: finishing write tid 3 to smithi19530025-49 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:39: finishing write tid 2 to smithi19530025-39 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: finishing write tid 4 to smithi19530025-49 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: oid 49 updating version 0 to 11 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: oid 49 updating version 11 to 13 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: oid 49 version 13 is already newer than 12 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 49 v 13 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-16T02:14:24.439 INFO:tasks.rados.rados.0.smithi195.stdout:50: finishing write tid 1 to smithi19530025-50 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:50: finishing write tid 2 to smithi19530025-50 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:39: finishing write tid 3 to smithi19530025-39 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:39: oid 39 updating version 0 to 3 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:39: oid 39 updating version 3 to 4 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 39 v 4 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:46: finishing write tid 3 to smithi19530025-46 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:46: oid 46 updating version 0 to 6 2024-09-16T02:14:24.440 INFO:tasks.rados.rados.0.smithi195.stdout:46: oid 46 version 6 is already newer than 5 2024-09-16T02:14:24.441 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T02:14:24.441 INFO:tasks.rados.rados.0.smithi195.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-16T02:14:24.443 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:24.443 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:24.459 INFO:tasks.rados.rados.0.smithi195.stdout:48: finishing write tid 3 to smithi19530025-48 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:48: finishing write tid 4 to smithi19530025-48 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:48: oid 48 updating version 0 to 14 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:48: oid 48 version 14 is already newer than 13 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:48: oid 48 updating version 14 to 15 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-16T02:14:24.461 INFO:tasks.rados.rados.0.smithi195.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-16T02:14:24.469 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:24.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:24.484 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.484 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:24.489 INFO:tasks.rados.rados.0.smithi195.stdout:25: finishing write tid 2 to smithi19530025-25 2024-09-16T02:14:24.492 INFO:tasks.rados.rados.0.smithi195.stdout:25: finishing write tid 3 to smithi19530025-25 2024-09-16T02:14:24.492 INFO:tasks.rados.rados.0.smithi195.stdout:25: oid 25 updating version 0 to 8 2024-09-16T02:14:24.492 INFO:tasks.rados.rados.0.smithi195.stdout:25: oid 25 version 8 is already newer than 7 2024-09-16T02:14:24.492 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 25 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T02:14:24.492 INFO:tasks.rados.rados.0.smithi195.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-16T02:14:24.495 INFO:tasks.rados.rados.0.smithi195.stdout:50: finishing write tid 3 to smithi19530025-50 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:31: finishing write tid 2 to smithi19530025-31 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:50: finishing write tid 4 to smithi19530025-50 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:50: oid 50 updating version 0 to 6 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:50: oid 50 updating version 6 to 7 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:50: oid 50 version 7 is already newer than 5 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 50 v 7 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-16T02:14:24.496 INFO:tasks.rados.rados.0.smithi195.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-16T02:14:24.498 INFO:tasks.rados.rados.0.smithi195.stdout:31: finishing write tid 3 to smithi19530025-31 2024-09-16T02:14:24.498 INFO:tasks.rados.rados.0.smithi195.stdout:31: oid 31 updating version 0 to 5 2024-09-16T02:14:24.498 INFO:tasks.rados.rados.0.smithi195.stdout:31: oid 31 updating version 5 to 6 2024-09-16T02:14:24.498 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T02:14:24.498 INFO:tasks.rados.rados.0.smithi195.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-16T02:14:24.499 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:14:24.508 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:24.502+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:24.610 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:24.604+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:24.668 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.669 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:24.682 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:14:24.711 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:24.711+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:24.812 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:24.811+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:24.834 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:24.835 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:24.851 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.851 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:24.854 INFO:tasks.rados.rados.0.smithi195.stdout:26: finishing write tid 2 to smithi19530025-26 2024-09-16T02:14:24.856 INFO:tasks.rados.rados.0.smithi195.stdout:25: done (15 left) 2024-09-16T02:14:24.856 INFO:tasks.rados.rados.0.smithi195.stdout:31: done (14 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:37: done (13 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:39: done (12 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:41: done (11 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:42: done (10 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:43: done (9 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:44: done (8 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:45: done (7 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:46: done (6 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:47: done (5 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:48: done (4 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:49: done (3 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:50: done (2 left) 2024-09-16T02:14:24.857 INFO:tasks.rados.rados.0.smithi195.stdout:52: done (1 left) 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:53: copy_from oid 4 from oid 49 current snap is 0 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:54: snap_create 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:26: finishing write tid 3 to smithi19530025-26 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:26: oid 26 updating version 0 to 6 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:26: oid 26 version 6 is already newer than 5 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-16T02:14:24.858 INFO:tasks.rados.rados.0.smithi195.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-16T02:14:24.868 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:14:24.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:24.869 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:24.875 INFO:tasks.rados.rados.0.smithi195.stdout:53: finishing copy_from racing read to smithi19530025-4 2024-09-16T02:14:24.876 INFO:tasks.rados.rados.0.smithi195.stdout:53: finishing copy_from to smithi19530025-4 2024-09-16T02:14:24.876 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 4 v 13 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T02:14:24.913 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:24.911+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:24.992 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:24.993 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.006 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:14:25.047 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:25.044+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:25.114 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:25.111+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:25.170 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.170 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.185 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:14:25.214 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:25.210+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:25.269 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:14:25.269 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T02:14:25.274 INFO:tasks.rados.rados.0.smithi195.stdout:26: done (2 left) 2024-09-16T02:14:25.274 INFO:tasks.rados.rados.0.smithi195.stdout:53: done (1 left) 2024-09-16T02:14:25.274 INFO:tasks.rados.rados.0.smithi195.stdout:54: done (0 left) 2024-09-16T02:14:25.274 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 30 current snap is 1 2024-09-16T02:14:25.274 INFO:tasks.rados.rados.0.smithi195.stdout:55: seq_num 51 ranges {491520=491520,983040=376832} 2024-09-16T02:14:25.280 INFO:tasks.rados.rados.0.smithi195.stdout:55: writing smithi19530025-30 from 491520 to 983040 tid 1 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:55: writing smithi19530025-30 from 983040 to 1359872 tid 2 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:56: copy_from oid 3 from oid 37 current snap is 1 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:57: rollback oid 17 current snap is 1 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 17 to 0 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:58: copy_from oid 36 from oid 6 current snap is 1 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:59: rollback oid 11 current snap is 1 2024-09-16T02:14:25.283 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 11 to 0 2024-09-16T02:14:25.284 INFO:tasks.rados.rados.0.smithi195.stdout:60: delete oid 4 current snap is 1 2024-09-16T02:14:25.295 INFO:tasks.rados.rados.0.smithi195.stdout:60: done (5 left) 2024-09-16T02:14:25.295 INFO:tasks.rados.rados.0.smithi195.stdout:61: read oid 1 snap -1 2024-09-16T02:14:25.295 INFO:tasks.rados.rados.0.smithi195.stdout:61: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T02:14:25.295 INFO:tasks.rados.rados.0.smithi195.stdout:62: delete oid 27 current snap is 1 2024-09-16T02:14:25.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T02:14:25.299 INFO:tasks.rados.rados.0.smithi195.stdout:55: finishing write tid 1 to smithi19530025-30 2024-09-16T02:14:25.306 INFO:tasks.rados.rados.0.smithi195.stdout:57: finishing rollback tid 0 to smithi19530025-17 2024-09-16T02:14:25.306 INFO:tasks.rados.rados.0.smithi195.stdout:61: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T02:14:25.306 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:25.307 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 3182 2024-09-16T02:14:25.309 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 22606 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:55: finishing write tid 2 to smithi19530025-30 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:55: finishing write tid 3 to smithi19530025-30 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:57: finishing rollback tid 1 to smithi19530025-17 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:59: finishing rollback tid 0 to smithi19530025-11 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:57: finishing rollback tid 2 to smithi19530025-17 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:59: finishing rollback tid 1 to smithi19530025-11 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:59: finishing rollback tid 2 to smithi19530025-11 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 19 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:57: done (6 left) 2024-09-16T02:14:25.310 INFO:tasks.rados.rados.0.smithi195.stdout:59: done (5 left) 2024-09-16T02:14:25.311 INFO:tasks.rados.rados.0.smithi195.stdout:61: done (4 left) 2024-09-16T02:14:25.311 INFO:tasks.rados.rados.0.smithi195.stdout:62: done (3 left) 2024-09-16T02:14:25.311 INFO:tasks.rados.rados.0.smithi195.stdout:63: delete oid 47 current snap is 1 2024-09-16T02:14:25.312 INFO:tasks.rados.rados.0.smithi195.stdout:56: finishing copy_from racing read to smithi19530025-3 2024-09-16T02:14:25.312 INFO:tasks.rados.rados.0.smithi195.stdout:58: finishing copy_from racing read to smithi19530025-36 2024-09-16T02:14:25.315 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:25.315+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:25.315 INFO:tasks.rados.rados.0.smithi195.stdout:55: finishing write tid 4 to smithi19530025-30 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:55: oid 30 updating version 0 to 16 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:55: oid 30 updating version 16 to 18 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:55: oid 30 version 18 is already newer than 17 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 18 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:55: left oid 30 (ObjNum 51 snap 1 seq_num 51) 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:56: finishing copy_from to smithi19530025-3 2024-09-16T02:14:25.316 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 3 v 4 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:55: done (3 left) 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:56: done (2 left) 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:63: done (1 left) 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:64: read oid 1 snap -1 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:64: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:65: rollback oid 39 current snap is 1 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 39 to 0 2024-09-16T02:14:25.317 INFO:tasks.rados.rados.0.smithi195.stdout:66: read oid 24 snap 0 2024-09-16T02:14:25.318 INFO:tasks.rados.rados.0.smithi195.stdout:66: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T02:14:25.318 INFO:tasks.rados.rados.0.smithi195.stdout:67: snap_create 2024-09-16T02:14:25.318 INFO:tasks.rados.rados.0.smithi195.stdout:58: finishing copy_from to smithi19530025-36 2024-09-16T02:14:25.318 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 36 v 2 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T02:14:25.323 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.323 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.324 INFO:tasks.rados.rados.0.smithi195.stdout:65: finishing rollback tid 0 to smithi19530025-39 2024-09-16T02:14:25.324 INFO:tasks.rados.rados.0.smithi195.stdout:64: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-16T02:14:25.324 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:25.325 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 25342 2024-09-16T02:14:25.325 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 17278 2024-09-16T02:14:25.325 INFO:tasks.rados.rados.0.smithi195.stdout:65: finishing rollback tid 1 to smithi19530025-39 2024-09-16T02:14:25.326 INFO:tasks.rados.rados.0.smithi195.stdout:65: finishing rollback tid 2 to smithi19530025-39 2024-09-16T02:14:25.326 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 39 v 11 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T02:14:25.327 INFO:tasks.rados.rados.0.smithi195.stdout:66: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-16T02:14:25.327 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:25.332 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 178446 2024-09-16T02:14:25.334 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:25.339 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:14:25.416 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:25.416+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:25.456 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.456 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:14:25.517 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:25.516+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:25.587 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.587 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.601 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:14:25.651 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:25.648+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:25.719 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:25.719+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:25.726 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.726 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:25.820 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:25.819+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:25.868 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:25.868 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:25.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:25.922 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:25.919+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:26.003 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.003 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:26.023 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:26.020+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:26.124 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:26.118+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:26.148 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.148 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:26.224 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:26.223+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:26.271 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:58: done (4 left) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:64: done (3 left) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:65: done (2 left) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:66: done (1 left) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:67: done (0 left) 2024-09-16T02:14:26.279 INFO:tasks.rados.rados.0.smithi195.stdout:68: snap_create 2024-09-16T02:14:26.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T02:14:26.325 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:26.324+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:26.334 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.335 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:26.426 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:26.427+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:26.513 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.513 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:26.529 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:26.528+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:26.630 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:26.628+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:26.648 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.648 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:26.731 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:26.726+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:26.780 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.780 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:26.832 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:26.828+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:26.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:26.933 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:26.932+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:26.961 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:26.961 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:26.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:27.033 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:27.026+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:27.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:27.053 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:27.078 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:27.079 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:27.096 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:27.096 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:27.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:27.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:27.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:27.134 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:27.135 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:27.135 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:27.132+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:27.158 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:27.158 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:27.189 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:27.189 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:27.217 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:27.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:27.235 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:27.232+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:27.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:27.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:27.254 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:27.254 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:27.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:14:27.276 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:27.276 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:27.281 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T02:14:27.292 INFO:tasks.rados.rados.0.smithi195.stdout:68: done (0 left) 2024-09-16T02:14:27.292 INFO:tasks.rados.rados.0.smithi195.stdout:69: read oid 34 snap -1 2024-09-16T02:14:27.292 INFO:tasks.rados.rados.0.smithi195.stdout:69: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-16T02:14:27.292 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 45 current snap is 3 2024-09-16T02:14:27.292 INFO:tasks.rados.rados.0.smithi195.stdout:70: seq_num 55 ranges {753664=557056,1310720=376832} 2024-09-16T02:14:27.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:14:27.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T02:14:27.306 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:27.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:27.313 INFO:tasks.rados.rados.0.smithi195.stdout:70: writing smithi19530025-45 from 753664 to 1310720 tid 1 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:70: writing smithi19530025-45 from 1310720 to 1687552 tid 2 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:71: rmattr oid 39 current snap is 3 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:71: done (2 left) 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:72: rollback oid 46 current snap is 3 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 46 to 0 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:73: copy_from oid 41 from oid 40 current snap is 3 2024-09-16T02:14:27.320 INFO:tasks.rados.rados.0.smithi195.stdout:69: expect (ObjNum 33 snap 0 seq_num 33) 2024-09-16T02:14:27.326 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 334502 2024-09-16T02:14:27.336 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:27.332+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:27.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:27.337 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:27.337 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:27.361 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 391344 2024-09-16T02:14:27.367 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:27.367 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:27.367 INFO:tasks.rados.rados.0.smithi195.stdout:70: finishing write tid 1 to smithi19530025-45 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:70: finishing write tid 2 to smithi19530025-45 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:69: done (3 left) 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:74: snap_create 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:70: finishing write tid 3 to smithi19530025-45 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:72: finishing rollback tid 0 to smithi19530025-46 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:72: finishing rollback tid 1 to smithi19530025-46 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:72: finishing rollback tid 2 to smithi19530025-46 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 10 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-16T02:14:27.368 INFO:tasks.rados.rados.0.smithi195.stdout:73: finishing copy_from racing read to smithi19530025-41 2024-09-16T02:14:27.369 INFO:tasks.rados.rados.0.smithi195.stdout:73: finishing copy_from to smithi19530025-41 2024-09-16T02:14:27.369 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 41 v 6 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:14:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:27.402 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:27.405 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:27.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:14:27.432 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:27.432 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:27.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:27.458+0000 7f08908f1640 0 -- 172.21.15.3:0/2079666734 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f088c21f120 msgr2=0x7f088c221510 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:27.463 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:27.458+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:27.471 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:27.512 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:27.512 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:27.538 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:27.536+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 35/32 ECSubWrite(tid=114, reqid=client.4598.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.292919+0000", 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2723504380000001, 2024-09-16T02:14:27.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021856860000000001, 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.292919+0000", 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.292919+0000", 2024-09-16T02:14:27.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.292920+0000", 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2899999999999999e-07 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.293196+0000", 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027585799999999997 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.293523+0000", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032773900000000001 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.293559+0000", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5203999999999997e-05 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.293634+0000", 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5624999999999995e-05 2024-09-16T02:14:27.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.293641+0000", 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4099999999999996e-06 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.295071+0000", 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001430057 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.295105+0000", 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4363999999999998e-05 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:26, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.298707+0000", 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2665632709999999, 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025861500000000002, 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298707+0000", 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298707+0000", 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298708+0000", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1829999999999999e-06 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298726+0000", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8652999999999999e-05 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298769+0000", 2024-09-16T02:14:27.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2914000000000003e-05 2024-09-16T02:14:27.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298779+0000", 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.91e-06 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298871+0000", 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1485999999999998e-05 2024-09-16T02:14:27.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.298883+0000", 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2279999999999999e-05 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301099+0000", 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022161870000000001 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301293+0000", 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000193537 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 35/32 ECSubWrite(tid=115, reqid=client.4598.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300233+0000", 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2650371969999998, 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031994380000000002, 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300233+0000", 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300233+0000", 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300234+0000", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4050000000000001e-06 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300256+0000", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2072000000000001e-05 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300400+0000", 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014431900000000001 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300408+0000", 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0399999999999993e-06 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300479+0000", 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0395000000000006e-05 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300488+0000", 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1940000000000004e-06 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.303396+0000", 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029082000000000001 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.303432+0000", 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5812999999999997e-05 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.582 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.4s1 35/32 ECSubWrite(tid=4, reqid=client.4598.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300274+0000", 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.264996209, 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031101950000000001, 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300274+0000", 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300274+0000", 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300274+0000", 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1399999999999997e-07 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300280+0000", 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1659999999999999e-06 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300435+0000", 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015514700000000001 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300440+0000", 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5770000000000004e-06 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300516+0000", 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5743999999999999e-05 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300527+0000", 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1294000000000001e-05 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.303342+0000", 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028147150000000002 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.303384+0000", 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2138000000000002e-05 2024-09-16T02:14:27.587 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/32 ECSubWrite(tid=2, reqid=client.4598.0:34, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300297+0000", 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2649727209999999, 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039847550000000004, 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.588 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300297+0000", 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300297+0000", 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300297+0000", 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4600000000000001e-07 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300357+0000", 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0099999999999997e-05 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300465+0000", 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107247 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300476+0000", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1456e-05 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300738+0000", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026244000000000001 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300746+0000", 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4120000000000004e-06 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304237+0000", 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003491423 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304282+0000", 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4530999999999998e-05 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.592 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:46 3.0s0 3:0952f51a:::smithi19530025-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2647306380000001, 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013865147, 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.593 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300541+0000", 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.193e-06 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301465+0000", 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092351300000000005 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301478+0000", 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.348e-05 2024-09-16T02:14:27.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301487+0000", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3869999999999995e-06 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301559+0000", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2444e-05 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301744+0000", 2024-09-16T02:14:27.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185019 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.302843+0000", 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010987550000000001 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305418+0000", 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025746219999999999 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314319+0000", 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089012259999999999 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314404+0000", 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5507999999999995e-05 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.598 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:47 3.0s0 3:0952f51a:::smithi19530025-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.263762378, 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013156407, 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 47 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301508+0000", 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079e-06 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301513+0000", 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.194e-06 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301521+0000", 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2309999999999996e-06 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301527+0000", 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4710000000000004e-06 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304040+0000", 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025124589999999999 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304279+0000", 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023908499999999999 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304869+0000", 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058981500000000004 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.307149+0000", 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022805849999999999 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314589+0000", 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074399310000000003 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314664+0000", 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:48 3.0s0 3:0952f51a:::smithi19530025-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.263576595, 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021785207000000001, 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 48 2024-09-16T02:14:27.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0359999999999999e-06 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301701+0000", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9700000000000002e-06 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301718+0000", 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7201999999999999e-05 2024-09-16T02:14:27.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301729+0000", 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0516999999999999e-05 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305321+0000", 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035922570000000002 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:27.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305405+0000", 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3998000000000005e-05 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314939+0000", 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095341579999999992 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.315042+0000", 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102858 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323478+0000", 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084362110000000008 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.316080+0000", 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2491893119999999, 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049377880000000002, 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.610 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316080+0000", 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316080+0000", 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316081+0000", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.569999999999999e-07 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316087+0000", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.604e-06 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316128+0000", 2024-09-16T02:14:27.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1417e-05 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.316140+0000", 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1661999999999999e-05 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320992+0000", 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004852365 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321018+0000", 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5783e-05 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.317567+0000", 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.247702785, 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035122970000000002, 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317567+0000", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317567+0000", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317568+0000", 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9999999999999995e-07 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317575+0000", 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5379999999999999e-06 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317618+0000", 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3101e-05 2024-09-16T02:14:27.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317627+0000", 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3239999999999992e-06 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321060+0000", 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034332680000000002 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321079+0000", 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9066e-05 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 35/32 ECSubWrite(tid=120, reqid=client.4598.0:55, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.317777+0000", 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.247492415, 2024-09-16T02:14:27.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021239509999999998, 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317777+0000", 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317777+0000", 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317778+0000", 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317956+0000", 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017758100000000001 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.317998+0000", 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2211e-05 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318006+0000", 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2279999999999998e-06 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318077+0000", 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1063999999999996e-05 2024-09-16T02:14:27.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318087+0000", 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0489999999999993e-06 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.319865+0000", 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001778806 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.319901+0000", 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6001999999999999e-05 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.318150+0000", 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2471201179999998, 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003059343, 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318150+0000", 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318150+0000", 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318151+0000", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8899999999999998e-07 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318156+0000", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9400000000000001e-06 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318209+0000", 2024-09-16T02:14:27.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3699000000000002e-05 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318217+0000", 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0830000000000004e-06 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321192+0000", 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029743719999999999 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321209+0000", 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7260000000000001e-05 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.318184+0000", 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2470852880000001, 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002954997, 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318184+0000", 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318184+0000", 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318185+0000", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9200000000000001e-07 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318189+0000", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9099999999999998e-06 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318238+0000", 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9283999999999998e-05 2024-09-16T02:14:27.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.318244+0000", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5539999999999997e-06 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321119+0000", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028757589999999999 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321139+0000", 2024-09-16T02:14:27.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9998000000000001e-05 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs3 35/32 ECSubWrite(tid=0, reqid=client.4598.0:49, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.319808+0000", 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2454622849999999, 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033436220000000001, 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.319808+0000", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.319808+0000", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.319808+0000", 2024-09-16T02:14:27.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8800000000000001e-07 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320196+0000", 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000387814 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320237+0000", 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0771e-05 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320246+0000", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2939999999999994e-06 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320344+0000", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7781000000000005e-05 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320353+0000", 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2240000000000001e-06 2024-09-16T02:14:27.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323108+0000", 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027550679999999998 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323151+0000", 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2781999999999998e-05 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.0s0 35/32 ECSubRead(tid=3, to_read={3:0952f51a:::smithi19530025-16:head=0,4096,0}, subchunks={3:0952f51a:::smithi19530025-16:head=[0,1]}, attrs_to_read=))", 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.320907+0000", 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2443627909999999, 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023035059999999999, 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320907+0000", 2024-09-16T02:14:27.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320907+0000", 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320907+0000", 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320907+0000", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320936+0000", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9496e-05 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.320947+0000", 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0360000000000001e-05 2024-09-16T02:14:27.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.321251+0000", 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030394599999999998 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323211+0000", 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001959704 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs1 35/32 ECSubWrite(tid=6, reqid=client.4598.0:70, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.338734+0000", 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.2265361330000002, 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024541630000000001, 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.338734+0000", 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.338734+0000", 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.338735+0000", 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.339230+0000", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049529299999999995 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.339278+0000", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7942000000000003e-05 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.339289+0000", 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1168e-05 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.339358+0000", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8823000000000002e-05 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.339368+0000", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6989999999999998e-06 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.341142+0000", 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001774797 2024-09-16T02:14:27.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.341188+0000", 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5503999999999997e-05 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 35/32 ECSubWrite(tid=8, reqid=client.4598.0:136, at_version=35'5, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.414219+0000", 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.1510510310000002, 2024-09-16T02:14:27.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028341629999999998, 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414219+0000", 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414219+0000", 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414220+0000", 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.384e-06 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414684+0000", 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046374600000000001 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414724+0000", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9894999999999997e-05 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414732+0000", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.371e-06 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414791+0000", 2024-09-16T02:14:27.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9259999999999998e-05 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.414798+0000", 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3729999999999998e-06 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417017+0000", 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002219483 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417053+0000", 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5651000000000001e-05 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:139 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.1486282010000002, 2024-09-16T02:14:27.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066880840000000004, 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416644+0000", 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9860000000000001e-06 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417552+0000", 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090839999999999996 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417565+0000", 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3406e-05 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417572+0000", 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2779999999999999e-06 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417622+0000", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0717999999999998e-05 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417725+0000", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102357 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.418463+0000", 2024-09-16T02:14:27.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073846399999999998 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419987+0000", 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001523735 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423245+0000", 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032577930000000001 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423330+0000", 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4946999999999998e-05 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:140 3.9s0 3:9a381d46:::smithi19530025-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.1476824749999999, 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060159289999999997, 2024-09-16T02:14:27.655 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417588+0000", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3200000000000005e-07 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417591+0000", 2024-09-16T02:14:27.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6800000000000002e-06 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417596+0000", 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8150000000000002e-06 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417600+0000", 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.337e-06 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419092+0000", 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014920630000000001 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419203+0000", 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110753 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419707+0000", 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050430500000000003 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.420177+0000", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047057700000000002 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423523+0000", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003345878 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423603+0000", 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9888999999999994e-05 2024-09-16T02:14:27.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:141 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 3.1476647149999999, 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073674559999999997, 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:27.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:27.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417608+0000", 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8899999999999999e-06 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:27.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417612+0000", 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0770000000000001e-06 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417616+0000", 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1389999999999999e-06 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419930+0000", 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023145040000000002 2024-09-16T02:14:27.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419980+0000", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0439000000000003e-05 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423911+0000", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039302770000000002 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424016+0000", 2024-09-16T02:14:27.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104734 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424973+0000", 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095702600000000004 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:27.666 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:27.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:14:27.669 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:27.667+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:27.770 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:27.766+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:27.800 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:27.800 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:27.800 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:27.800 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:27.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:14:27.871 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:27.868+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:27.936 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:27.936 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:27.936 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:27.936 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:27.936 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:27.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:14:27.971 INFO:tasks.rados.rados.0.smithi195.stdout:70: finishing write tid 4 to smithi19530025-45 2024-09-16T02:14:27.971 INFO:tasks.rados.rados.0.smithi195.stdout:70: oid 45 updating version 0 to 10 2024-09-16T02:14:27.972 INFO:tasks.rados.rados.0.smithi195.stdout:70: oid 45 updating version 10 to 12 2024-09-16T02:14:27.972 INFO:tasks.rados.rados.0.smithi195.stdout:70: oid 45 version 12 is already newer than 11 2024-09-16T02:14:27.972 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 12 (ObjNum 55 snap 3 seq_num 55) dirty exists 2024-09-16T02:14:27.972 INFO:tasks.rados.rados.0.smithi195.stdout:70: left oid 45 (ObjNum 55 snap 3 seq_num 55) 2024-09-16T02:14:27.972 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:27.966+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:28.073 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:28.066+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:28.075 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.075 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.757737785, 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051833920000000002, 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318273+0000", 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4999999999999998e-07 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318481+0000", 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020832199999999999 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318589+0000", 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107645 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318612+0000", 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3468999999999998e-05 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318803+0000", 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019100200000000001 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318832+0000", 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8812000000000001e-05 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323396+0000", 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045639950000000004 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323455+0000", 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9697000000000001e-05 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=17, reqid=client.4598.0:193, at_version=38'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.324080+0000", 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.75192999199999999, 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020626120000000001, 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324080+0000", 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324080+0000", 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324081+0000", 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.294e-06 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324235+0000", 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000153413 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324333+0000", 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8133000000000004e-05 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324349+0000", 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5795999999999999e-05 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324466+0000", 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117782 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324481+0000", 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4633e-05 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326056+0000", 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015746250000000001 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326142+0000", 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6935999999999996e-05 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=18, reqid=client.4598.0:194, at_version=38'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.327105+0000", 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74890513299999995, 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017625939999999999, 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327105+0000", 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327105+0000", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327106+0000", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7370000000000001e-06 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327147+0000", 2024-09-16T02:14:28.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0145000000000003e-05 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327242+0000", 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4919000000000004e-05 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327257+0000", 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.501e-05 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327462+0000", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020559800000000001 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327492+0000", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0185e-05 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328795+0000", 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013031200000000001 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328867+0000", 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1879999999999996e-05 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=19, reqid=client.4598.0:194, at_version=0'0, trim_to=0'0, roll_forward_to=38'12))", 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.334254+0000", 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.74175595100000002, 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00154013, 2024-09-16T02:14:28.090 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334254+0000", 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334254+0000", 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334256+0000", 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.883e-06 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334278+0000", 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2429000000000002e-05 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334356+0000", 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7733999999999998e-05 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334372+0000", 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5979000000000001e-05 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334575+0000", 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020280499999999999 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334592+0000", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7699000000000001e-05 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335717+0000", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011246170000000001 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335794+0000", 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6983999999999993e-05 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.094 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.095 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.095 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.095 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:14:28.174 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:28.172+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:28.219 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.219 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:28.220 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:28.220 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.233 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:14:28.275 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:28.272+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:28.295 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:14:28.296 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T02:14:28.301 INFO:tasks.rados.rados.0.smithi195.stdout:70: done (3 left) 2024-09-16T02:14:28.302 INFO:tasks.rados.rados.0.smithi195.stdout:72: done (2 left) 2024-09-16T02:14:28.302 INFO:tasks.rados.rados.0.smithi195.stdout:73: done (1 left) 2024-09-16T02:14:28.302 INFO:tasks.rados.rados.0.smithi195.stdout:74: done (0 left) 2024-09-16T02:14:28.302 INFO:tasks.rados.rados.0.smithi195.stdout:75: snap_create 2024-09-16T02:14:28.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T02:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:28.358 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:14:28.418 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:28.419+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:28.477 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:28.476+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:28.496 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.496 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:28.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:28.496 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:28.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4293.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.325911283, 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084256020000000008, 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4293", 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168352+0000", 2024-09-16T02:14:28.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0299999999999999e-06 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168366+0000", 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168399+0000", 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2588e-05 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168427+0000", 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8371e-05 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168562+0000", 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134337 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168584+0000", 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2011999999999999e-05 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173264+0000", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046804209999999997 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173570+0000", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000305638 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173890+0000", 2024-09-16T02:14:28.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320122 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176184+0000", 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022945679999999999 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176505+0000", 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032044099999999998 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176650+0000", 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145253 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176771+0000", 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012114500000000001 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.503 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:192 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~557056] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1794475769999999, 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015517632999999999, 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:28.504 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 192 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314812+0000", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029999999999999e-06 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315788+0000", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097551599999999995 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315825+0000", 2024-09-16T02:14:28.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6742000000000002e-05 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315847+0000", 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2396e-05 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316018+0000", 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170729 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316179+0000", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016178000000000001 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318923+0000", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002743381 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:14:28.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071732469999999998 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330202+0000", 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004106391 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330327+0000", 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012444800000000001 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:193 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1728449990000001, 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094258069999999996, 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321415+0000", 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8789999999999999e-06 2024-09-16T02:14:28.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321889+0000", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047466400000000002 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321924+0000", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4134999999999998e-05 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321944+0000", 2024-09-16T02:14:28.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0060000000000001e-05 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322086+0000", 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014260999999999999 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322329+0000", 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024320400000000001 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324137+0000", 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018075879999999999 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328050+0000", 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039127119999999996 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330645+0000", 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025952829999999999 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330838+0000", 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019267199999999999 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:194 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (56) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1722685529999999, 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011982552, 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 194 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321990+0000", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.756e-06 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322010+0000", 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9443e-05 2024-09-16T02:14:28.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322033+0000", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322049+0000", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6201999999999998e-05 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326191+0000", 2024-09-16T02:14:28.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041419430000000004 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326397+0000", 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020605800000000001 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327141+0000", 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074450000000000004 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328976+0000", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018343109999999999 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333836+0000", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048599389999999998 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333971+0000", 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013532799999999999 2024-09-16T02:14:28.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 37/32 ECSubWriteReply(tid=16, last_complete=38'10, committed=1, applied=1))", 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322000+0000", 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1722568840000001, 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003749432, 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322000+0000", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322000+0000", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322001+0000", 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3459999999999999e-06 2024-09-16T02:14:28.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322011+0000", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9869999999999994e-06 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322089+0000", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.729e-05 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322106+0000", 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7638999999999998e-05 2024-09-16T02:14:28.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325695+0000", 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003588866 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325749+0000", 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4304e-05 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:195 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.172174067, 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014685304, 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 195 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322084+0000", 2024-09-16T02:14:28.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322092+0000", 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6529999999999994e-06 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322111+0000", 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8909000000000001e-05 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322124+0000", 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3589e-05 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327750+0000", 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056258819999999996 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327883+0000", 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013255499999999999 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334789+0000", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069061050000000001 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335013+0000", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022373400000000001 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.336768+0000", 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001755454 2024-09-16T02:14:28.529 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=16, last_complete=38'10, committed=1, applied=1))", 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322177+0000", 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1720802180000001, 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037331729999999998, 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322177+0000", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322177+0000", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322178+0000", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1200000000000001e-06 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322185+0000", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5830000000000001e-06 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322255+0000", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9696999999999994e-05 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322271+0000", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6252999999999998e-05 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325859+0000", 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035874219999999998 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325910+0000", 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1097999999999998e-05 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=16, last_complete=38'10, committed=1, applied=1))", 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.323572+0000", 2024-09-16T02:14:28.533 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1706847389999999, 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002527999, 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323572+0000", 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323572+0000", 2024-09-16T02:14:28.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323574+0000", 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.776e-06 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323584+0000", 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0594000000000001e-05 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323659+0000", 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4252999999999995e-05 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323672+0000", 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3593e-05 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326003+0000", 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023306999999999998 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326100+0000", 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7083000000000005e-05 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=17, last_complete=38'11, committed=1, applied=1))", 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.325992+0000", 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1682645979999999, 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002060451, 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.537 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325992+0000", 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325992+0000", 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.325994+0000", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7799999999999999e-06 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326007+0000", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8706000000000001e-05 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326116+0000", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.039e-05 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327999+0000", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018833420000000001 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328053+0000", 2024-09-16T02:14:28.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3334000000000001e-05 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=17, last_complete=38'11, committed=1, applied=1))", 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.326154+0000", 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1681029789999999, 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020603190000000001, 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.541 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326154+0000", 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326154+0000", 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326156+0000", 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0600000000000002e-06 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326168+0000", 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2393e-05 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326255+0000", 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7122999999999999e-05 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326270+0000", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328157+0000", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018867389999999999 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328214+0000", 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7327999999999997e-05 2024-09-16T02:14:28.544 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=17, last_complete=38'11, committed=1, applied=1))", 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.326277+0000", 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1679795900000001, 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002103129, 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326277+0000", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326277+0000", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326279+0000", 2024-09-16T02:14:28.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326287+0000", 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3359999999999995e-06 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326346+0000", 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9134e-05 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326358+0000", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1749999999999999e-05 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328321+0000", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019628219999999999 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328380+0000", 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9787000000000002e-05 2024-09-16T02:14:28.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=17, last_complete=38'11, committed=1, applied=1))", 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.327474+0000", 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1667829329999999, 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001046017, 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327474+0000", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327474+0000", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327475+0000", 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4839999999999999e-06 2024-09-16T02:14:28.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327490+0000", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4099e-05 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327545+0000", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5213999999999997e-05 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327558+0000", 2024-09-16T02:14:28.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328473+0000", 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091443500000000001 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328520+0000", 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7311999999999997e-05 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=17, last_complete=38'11, committed=1, applied=1))", 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.329962+0000", 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1642947260000001, 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090320299999999999, 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.329962+0000", 2024-09-16T02:14:28.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.329962+0000", 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.329962+0000", 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4700000000000002e-07 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.329965+0000", 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6230000000000001e-06 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330050+0000", 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5062999999999995e-05 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330059+0000", 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4670000000000004e-06 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330434+0000", 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037579399999999999 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330865+0000", 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043090899999999999 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.556 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 38/32 ECSubWriteReply(tid=18, last_complete=38'12, committed=1, applied=1))", 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.333567+0000", 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1606893810000001, 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011342360000000001, 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333567+0000", 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333567+0000", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333568+0000", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4000000000000001e-07 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333573+0000", 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8060000000000002e-06 2024-09-16T02:14:28.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333620+0000", 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6978000000000003e-05 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333629+0000", 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6629999999999999e-06 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333711+0000", 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2373000000000006e-05 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334702+0000", 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00099077600000000007 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.11.22:1 3.8s0 3:165ce5f6:::smithi19530025-40:head [assert-version v6,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:28.560 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.340056+0000", 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.154201093, 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035641240000000001, 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340056+0000", 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340056+0000", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340058+0000", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7540000000000001e-06 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340066+0000", 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4759999999999998e-06 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340091+0000", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4675999999999999e-05 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340102+0000", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1009000000000001e-05 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340289+0000", 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018770599999999999 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.340440+0000", 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015079700000000001 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.343620+0000", 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003179706 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.8s0 38/32 ECSubReadReply(tid=21, attrs_read=0))", 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.341864+0000", 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1523926200000001, 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001805552, 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341864+0000", 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341864+0000", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341887+0000", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2399e-05 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342532+0000", 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00064490000000000001 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342595+0000", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3278e-05 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342605+0000", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0582e-05 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342769+0000", 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016369400000000001 2024-09-16T02:14:28.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.343670+0000", 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090069900000000003 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.8s0 38/32 ECSubReadReply(tid=21, attrs_read=0))", 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.341872+0000", 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1523850609999999, 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074427, 2024-09-16T02:14:28.568 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341872+0000", 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341872+0000", 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341879+0000", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.72e-06 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342320+0000", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044195900000000002 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342389+0000", 2024-09-16T02:14:28.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8449999999999997e-05 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342401+0000", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.201e-05 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342515+0000", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113585 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342616+0000", 2024-09-16T02:14:28.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010154600000000001 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpReadReply(3.8s0 38/32 ECSubReadReply(tid=21, attrs_read=0))", 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.341924+0000", 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1523331029999999, 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00079613800000000005, 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341924+0000", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341924+0000", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.341925+0000", 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.53e-06 2024-09-16T02:14:28.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342383+0000", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045721500000000002 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342458+0000", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5372999999999998e-05 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342476+0000", 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8033000000000002e-05 2024-09-16T02:14:28.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342653+0000", 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017713599999999999 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.342720+0000", 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6851000000000001e-05 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1445991250000001, 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045915540000000003, 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349659+0000", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0899999999999999e-06 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350269+0000", 2024-09-16T02:14:28.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061005800000000002 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350367+0000", 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8338999999999995e-05 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350395+0000", 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7313000000000001e-05 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350521+0000", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126945 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350539+0000", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7419e-05 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354142+0000", 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036026909999999999 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354249+0000", 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107699 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/32 ECSubWrite(tid=31, reqid=client.4598.0:199, at_version=0'0, trim_to=0'0, roll_forward_to=38'21))", 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.356515+0000", 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1377423580000001, 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001429844, 2024-09-16T02:14:28.580 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356515+0000", 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356515+0000", 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356516+0000", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7999999999999999e-06 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356538+0000", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1401999999999999e-05 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356608+0000", 2024-09-16T02:14:28.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0647000000000003e-05 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356623+0000", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4853e-05 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356750+0000", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012642799999999999 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356764+0000", 2024-09-16T02:14:28.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4222999999999999e-05 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.357861+0000", 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010974579999999999 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.357944+0000", 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3033000000000003e-05 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:28.585 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:28.585 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:14:28.587 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:28.578+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:28.687 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:28.686+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:28.757 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.757 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:28.757 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:28.757 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:14:28.788 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:28.788+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:28.889 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:28.888+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:28.939 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:28.939 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:28.939 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:28.939 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:28.939 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:28.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:14:28.998 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:28.998+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:29.070 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:29.070 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:29.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:29.070 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7901868920000004, 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:14:29.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7859644460000004, 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:14:29.075 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:14:29.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:14:29.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7856691370000002, 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006345626, 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283859+0000", 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283921+0000", 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1947999999999997e-05 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284076+0000", 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015468599999999999 2024-09-16T02:14:29.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284095+0000", 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8686e-05 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285435+0000", 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001340639 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285464+0000", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8411999999999998e-05 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290127+0000", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046638529999999999 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290204+0000", 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079000000000001e-05 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=1, reqid=client.4598.0:17, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7836355289999997, 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048188320000000003, 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285894+0000", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.667e-06 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285942+0000", 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8133999999999997e-05 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286032+0000", 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9883000000000007e-05 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286080+0000", 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7605000000000002e-05 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286237+0000", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015714200000000001 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286262+0000", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5012e-05 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043960889999999997 2024-09-16T02:14:29.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290710+0000", 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2299999999999997e-05 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.089 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7732125139999999, 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:14:29.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7705481340000002, 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.094 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:14:29.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7704889139999995, 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.099 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:14:29.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.770081609, 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7682008639999998, 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:14:29.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7679252999999999, 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:14:29.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7665999030000004, 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00583893, 2024-09-16T02:14:29.118 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302929+0000", 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.717e-06 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302970+0000", 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1564999999999999e-05 2024-09-16T02:14:29.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303048+0000", 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8059000000000006e-05 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303070+0000", 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282999999999999e-05 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306196+0000", 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00312663 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306213+0000", 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6747000000000001e-05 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308730+0000", 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002516796 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308766+0000", 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6133000000000001e-05 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=6, reqid=client.4598.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7651211499999997, 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045402039999999999, 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.123 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304408+0000", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3599999999999999e-06 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304438+0000", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9399000000000001e-05 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304525+0000", 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6935999999999996e-05 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304548+0000", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3272e-05 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304730+0000", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018221900000000001 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304756+0000", 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5908999999999999e-05 2024-09-16T02:14:29.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308884+0000", 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041276189999999999 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308946+0000", 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2490000000000006e-05 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=4, reqid=client.4598.0:52, at_version=35'3, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.311102+0000", 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7584247030000002, 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003723872, 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311102+0000", 2024-09-16T02:14:29.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311102+0000", 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311105+0000", 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.503e-06 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311803+0000", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069840099999999997 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311899+0000", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5733e-05 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.311923+0000", 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3538999999999999e-05 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312171+0000", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024804900000000002 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312194+0000", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3856e-05 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.314751+0000", 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002556769 2024-09-16T02:14:29.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.314826+0000", 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5022000000000001e-05 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=5, reqid=client.4598.0:53, at_version=35'4, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:29.132 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.312763+0000", 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7567638819999996, 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027230269999999998, 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312763+0000", 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312763+0000", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312765+0000", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9800000000000001e-06 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312808+0000", 2024-09-16T02:14:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3062000000000002e-05 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312889+0000", 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1181999999999996e-05 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.312930+0000", 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.314290+0000", 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00136082 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.314313+0000", 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2566999999999999e-05 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315398+0000", 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010853659999999999 2024-09-16T02:14:29.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315486+0000", 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7923999999999994e-05 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=8, reqid=client.4598.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.317261+0000", 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7522664419999998, 2024-09-16T02:14:29.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023251669999999999, 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317261+0000", 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317261+0000", 2024-09-16T02:14:29.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317263+0000", 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9199999999999998e-06 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317286+0000", 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3830999999999999e-05 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317359+0000", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2235000000000002e-05 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317376+0000", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7456e-05 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317507+0000", 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013127899999999999 2024-09-16T02:14:29.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.317525+0000", 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8114e-05 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.319506+0000", 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019804200000000001 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.319586+0000", 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9912000000000003e-05 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:64, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.322648+0000", 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7468793690000002, 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025533460000000002, 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.142 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322648+0000", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322648+0000", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322650+0000", 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2259999999999999e-06 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322930+0000", 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028038800000000001 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323028+0000", 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7998000000000006e-05 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323051+0000", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2733000000000001e-05 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323166+0000", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114944 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323176+0000", 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0502e-05 2024-09-16T02:14:29.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.325125+0000", 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019487630000000001 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.325201+0000", 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5791999999999995e-05 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=11, reqid=client.4598.0:65, at_version=35'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.323755+0000", 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.7457721030000002, 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023307359999999999, 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323755+0000", 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323755+0000", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323757+0000", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9530000000000002e-06 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323799+0000", 2024-09-16T02:14:29.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1854999999999998e-05 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323895+0000", 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6325999999999999e-05 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323918+0000", 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3132000000000001e-05 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324108+0000", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018996200000000001 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324140+0000", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2002000000000003e-05 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326023+0000", 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001883121 2024-09-16T02:14:29.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326086+0000", 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2385000000000007e-05 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.7193936519999999, 2024-09-16T02:14:29.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046039000000000002, 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350134+0000", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350792+0000", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065815600000000002 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350844+0000", 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1777999999999998e-05 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350861+0000", 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6826000000000001e-05 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351025+0000", 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164623 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351055+0000", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9672999999999999e-05 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354677+0000", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036216830000000001 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354737+0000", 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0534e-05 2024-09-16T02:14:29.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/32 ECSubWrite(tid=31, reqid=client.4598.0:199, at_version=0'0, trim_to=0'0, roll_forward_to=38'21))", 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.356589+0000", 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.7129378420000001, 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001361384, 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.156 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356589+0000", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356589+0000", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356591+0000", 2024-09-16T02:14:29.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9800000000000001e-06 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356614+0000", 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3278999999999999e-05 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356688+0000", 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3337000000000006e-05 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356703+0000", 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5063e-05 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356817+0000", 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113726 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.356831+0000", 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3925e-05 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.357887+0000", 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010565710000000001 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.357951+0000", 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3503000000000005e-05 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.160 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.161 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:29.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:14:29.163 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:29.160+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:29.264 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:29.260+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:29.298 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T02:14:29.304 INFO:tasks.rados.rados.0.smithi195.stdout:75: done (0 left) 2024-09-16T02:14:29.304 INFO:tasks.rados.rados.0.smithi195.stdout:76: snap_remove snap 4 2024-09-16T02:14:29.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T02:14:29.333 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:29.334 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:29.334 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:29.334 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:29.348 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:14:29.365 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:29.364+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:29.466 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:29.460+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:29.466 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:29.466 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:29.467 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:29.467 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:29.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:14:29.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:29.566 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:29.566+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:29.604 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:29.604 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.532333154, 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:29.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:14:29.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:14:29.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:50 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.027724309, 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019551556000000001, 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 50 2024-09-16T02:14:29.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575871+0000", 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575872+0000", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5700000000000002e-07 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575960+0000", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8003000000000004e-05 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575971+0000", 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1209000000000001e-05 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575979+0000", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8800000000000008e-06 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591098+0000", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015119085000000001 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591336+0000", 2024-09-16T02:14:29.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000238426 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.591731+0000", 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000394046 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594175+0000", 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024441599999999999 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595149+0000", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097475000000000003 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595306+0000", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000156918 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595423+0000", 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116322 2024-09-16T02:14:29.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:51 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.027601171000001, 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019744136999999998, 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 51 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575994+0000", 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.575995+0000", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3300000000000001e-07 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576002+0000", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9190000000000002e-06 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576012+0000", 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9690000000000006e-06 2024-09-16T02:14:29.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576020+0000", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5839999999999997e-06 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.592718+0000", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016698369000000001 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.592850+0000", 2024-09-16T02:14:29.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131877 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.593052+0000", 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020253799999999999 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595451+0000", 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023987660000000001 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595533+0000", 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2224000000000004e-05 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595617+0000", 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4136999999999998e-05 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595738+0000", 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120921 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.027563087000001, 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020204412000000001, 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.624 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576033+0000", 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6599999999999995e-07 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576055+0000", 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1931999999999999e-05 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576066+0000", 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1369000000000001e-05 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576074+0000", 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5630000000000003e-06 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594265+0000", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018191161000000001 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594479+0000", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021409700000000001 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594685+0000", 2024-09-16T02:14:29.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000205708 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595917+0000", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232564 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596058+0000", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014070900000000001 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596135+0000", 2024-09-16T02:14:29.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6878000000000005e-05 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596237+0000", 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101665 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:29.631 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.806256557000001, 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019987173, 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797340+0000", 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797353+0000", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3156000000000001e-05 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797372+0000", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8382000000000001e-05 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797386+0000", 2024-09-16T02:14:29.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4262e-05 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811114+0000", 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013728557000000001 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811354+0000", 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023977499999999999 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811632+0000", 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027806899999999998 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815429+0000", 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037971509999999999 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817182+0000", 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001752382 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817262+0000", 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0240000000000004e-05 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817326+0000", 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4022000000000005e-05 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.806185074999998, 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020141841000000001, 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.638 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797412+0000", 2024-09-16T02:14:29.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.243e-06 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797542+0000", 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077400000000001 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797562+0000", 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9644e-05 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797577+0000", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812322+0000", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014744597999999999 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812456+0000", 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134466 2024-09-16T02:14:29.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812671+0000", 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000214537 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815454+0000", 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027836179999999999 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817415+0000", 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019603979999999999 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817490+0000", 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5012000000000006e-05 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817552+0000", 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2331999999999999e-05 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.805992369999998, 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020132852, 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.644 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797605+0000", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7069999999999999e-06 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797620+0000", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5585000000000001e-05 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797639+0000", 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8212000000000001e-05 2024-09-16T02:14:29.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797658+0000", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9514999999999999e-05 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813882+0000", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016224045999999999 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814108+0000", 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022626600000000001 2024-09-16T02:14:29.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814462+0000", 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035324400000000001 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815978+0000", 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015162000000000001 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817614+0000", 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001636366 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817684+0000", 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9701000000000002e-05 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817736+0000", 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2009999999999998e-05 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.649 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.694518189, 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019924002999999999, 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.650 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 122 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909078+0000", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.23e-07 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909084+0000", 2024-09-16T02:14:29.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000002e-06 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909094+0000", 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8330000000000006e-06 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909104+0000", 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0781999999999999e-05 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922447+0000", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013342309 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922571+0000", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012409699999999999 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922765+0000", 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000193869 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926685+0000", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003919952 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928862+0000", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002177216 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928940+0000", 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8281000000000005e-05 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.929001+0000", 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1030999999999999e-05 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.694477349, 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:14:29.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.694352954999999, 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.662 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:29.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:1 3.3s0 3:c481b30a:::smithi19530025-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:29.668 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3262003179999997, 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021556188, 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277399+0000", 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4400000000000001e-06 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277527+0000", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012811100000000001 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277557+0000", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0645000000000002e-05 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277594+0000", 2024-09-16T02:14:29.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6848999999999999e-05 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277805+0000", 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021101299999999999 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.278199+0000", 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039380300000000002 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.280065+0000", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018661680000000001 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286512+0000", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064472219999999998 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298756+0000", 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012244026 2024-09-16T02:14:29.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298951+0000", 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194911 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:2 3.3s0 3:c481b30a:::smithi19530025-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3259464489999999, 2024-09-16T02:14:29.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022809754000000002, 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:29.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277650+0000", 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277657+0000", 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277675+0000", 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8148999999999999e-05 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277691+0000", 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5943e-05 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282023+0000", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004332247 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282308+0000", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028409399999999998 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285666+0000", 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033580379999999998 2024-09-16T02:14:29.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.292774+0000", 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071080379999999997 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300235+0000", 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074610759999999996 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300459+0000", 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224033 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3257817769999996, 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.680 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:14:29.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:14:29.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.685 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3223033839999996, 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:14:29.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:14:29.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.691 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3211812109999999, 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853072, 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282416+0000", 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282453+0000", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7261000000000001e-05 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298874+0000", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016420846999999999 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298893+0000", 2024-09-16T02:14:29.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9695e-05 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299501+0000", 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000607509 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299522+0000", 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1039e-05 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303194+0000", 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036718020000000001 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303267+0000", 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3598999999999998e-05 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.696 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3206603750000001, 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:14:29.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:14:29.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.702 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3205641559999997, 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:14:29.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:14:29.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.708 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:58 3.2s0 3:431ee5ca:::smithi19530025-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.288534802, 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021031579000000002, 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.709 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 58 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315062+0000", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315685+0000", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062327200000000002 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315711+0000", 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5519000000000001e-05 2024-09-16T02:14:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315731+0000", 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9775e-05 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322701+0000", 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069698679999999997 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323044+0000", 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034300800000000001 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324822+0000", 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017778659999999999 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.329203+0000", 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043815149999999999 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.335988+0000", 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067852160000000002 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.336092+0000", 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103927 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:59 3.2s0 3:431ee5ca:::smithi19530025-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.714 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2878249010000005, 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022592232, 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 59 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8729999999999999e-06 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315782+0000", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2629999999999992e-06 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315822+0000", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0095000000000002e-05 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315843+0000", 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0905999999999999e-05 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326893+0000", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011050249999999999 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.327039+0000", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145723 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328011+0000", 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097276899999999996 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.330713+0000", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027020989999999999 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338142+0000", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074284329999999999 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338363+0000", 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022082100000000001 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.719 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2877257350000004, 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:14:29.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:14:29.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:29.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.724 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:29.725 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:29.725 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:29.726 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:29.726 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:29.727 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:29.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:29.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:14:29.728 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:29.724+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:29.738 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:29.738 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:29.764 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:29.764 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:29.791 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:29.791 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:29.818 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:29.818 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:29.827 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:29.824+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:29.848 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:29.849 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:29.857 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:29.857 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:29.871 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:14:29.880 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:29.881 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:29.911 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:29.911 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:29.927 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:29.922+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:29.946 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:29.946 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:29.976 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:29.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:30.011 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:30.011 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:30.028 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:30.022+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:30.039 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.039 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:30.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:30.053 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:14:30.081 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:30.082 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:30.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:30.110+0000 7fdfd67fc640 0 -- 172.21.15.3:0/3547714694 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fdfe0221230 msgr2=0x7fdfe0223620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:30.122 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:30.129 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:30.122+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:30.156 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:30.156 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:30.179 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.179 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.193 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:14:30.237 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:30.236+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:30.307 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T02:14:30.313 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.313 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.318 INFO:tasks.rados.rados.0.smithi195.stdout:76: done (0 left) 2024-09-16T02:14:30.318 INFO:tasks.rados.rados.0.smithi195.stdout:77: setattr oid 45 current snap is 5 2024-09-16T02:14:30.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:14:30.321 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 1 current snap is 5 2024-09-16T02:14:30.321 INFO:tasks.rados.rados.0.smithi195.stdout:78: seq_num 56 ranges {32768=573440,606208=376832} 2024-09-16T02:14:30.329 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:14:30.331 INFO:tasks.rados.rados.0.smithi195.stdout:78: writing smithi19530025-1 from 32768 to 606208 tid 1 2024-09-16T02:14:30.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T02:14:30.339 INFO:tasks.rados.rados.0.smithi195.stdout:78: writing smithi19530025-1 from 606208 to 983040 tid 2 2024-09-16T02:14:30.339 INFO:tasks.rados.rados.0.smithi195.stdout:79: read oid 10 snap -1 2024-09-16T02:14:30.339 INFO:tasks.rados.rados.0.smithi195.stdout:79: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T02:14:30.340 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 14 (ObjNum 55 snap 3 seq_num 55) dirty exists 2024-09-16T02:14:30.340 INFO:tasks.rados.rados.0.smithi195.stdout:77: done (2 left) 2024-09-16T02:14:30.340 INFO:tasks.rados.rados.0.smithi195.stdout:80: copy_from oid 26 from oid 39 current snap is 5 2024-09-16T02:14:30.340 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 44 current snap is 5 2024-09-16T02:14:30.340 INFO:tasks.rados.rados.0.smithi195.stdout:81: seq_num 57 ranges {737280=589824,1327104=376832} 2024-09-16T02:14:30.362 INFO:tasks.rados.rados.0.smithi195.stdout:81: writing smithi19530025-44 from 737280 to 1327104 tid 1 2024-09-16T02:14:30.369 INFO:tasks.rados.rados.0.smithi195.stdout:81: writing smithi19530025-44 from 1327104 to 1703936 tid 2 2024-09-16T02:14:30.369 INFO:tasks.rados.rados.0.smithi195.stdout:82: copy_from oid 32 from oid 2 current snap is 5 2024-09-16T02:14:30.369 INFO:tasks.rados.rados.0.smithi195.stdout:83: read oid 29 snap -1 2024-09-16T02:14:30.369 INFO:tasks.rados.rados.0.smithi195.stdout:83: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T02:14:30.370 INFO:tasks.rados.rados.0.smithi195.stdout:84: read oid 19 snap -1 2024-09-16T02:14:30.370 INFO:tasks.rados.rados.0.smithi195.stdout:84: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T02:14:30.370 INFO:tasks.rados.rados.0.smithi195.stdout:85: snap_remove snap 2 2024-09-16T02:14:30.373 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:30.372+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:30.431 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:30.426+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:30.499 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.499 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.513 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:14:30.561 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:30.560+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:30.630 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.630 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.633 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:30.626+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:30.644 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:14:30.734 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:30.734+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:30.807 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.807 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.821 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:14:30.835 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:30.834+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:30.937 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:30.930+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:30.946 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:30.946 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:30.960 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:14:31.037 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:31.036+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:31.079 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.079 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.095 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:14:31.138 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:31.134+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:31.239 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:31.232+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:31.260 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.260 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.275 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:14:31.316 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:14:31.317 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T02:14:31.323 INFO:tasks.rados.rados.0.smithi195.stdout:85: done (7 left) 2024-09-16T02:14:31.324 INFO:tasks.rados.rados.0.smithi195.stdout:86: read oid 38 snap -1 2024-09-16T02:14:31.324 INFO:tasks.rados.rados.0.smithi195.stdout:86: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T02:14:31.324 INFO:tasks.rados.rados.0.smithi195.stdout:79: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-16T02:14:31.324 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:31.331 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 42886 2024-09-16T02:14:31.337 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:31.339 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:31.334+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:31.344 INFO:tasks.rados.rados.0.smithi195.stdout:78: finishing write tid 1 to smithi19530025-1 2024-09-16T02:14:31.344 INFO:tasks.rados.rados.0.smithi195.stdout:78: finishing write tid 2 to smithi19530025-1 2024-09-16T02:14:31.344 INFO:tasks.rados.rados.0.smithi195.stdout:79: done (7 left) 2024-09-16T02:14:31.344 INFO:tasks.rados.rados.0.smithi195.stdout:87: rollback oid 49 current snap is 5 2024-09-16T02:14:31.344 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 49 to 0 2024-09-16T02:14:31.345 INFO:tasks.rados.rados.0.smithi195.stdout:88: snap_remove snap 3 2024-09-16T02:14:31.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T02:14:31.404 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.404 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.418 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:14:31.441 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:31.436+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:31.541 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.541 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.541 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:31.534+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:31.555 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:14:31.642 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:31.634+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:31.688 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.688 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.702 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:14:31.743 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:31.742+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:31.822 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.823 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.837 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:14:31.844 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:31.844+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:31.946 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:31.944+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:31.955 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:31.956 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:31.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:14:32.047 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:32.046+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:32.091 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:32.092 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:32.106 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:14:32.148 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:32.144+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:32.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:32.227 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:32.227 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:32.241 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:14:32.249 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:32.248+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:32.317 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:88: done (8 left) 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:89: read oid 4 snap 0 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:89: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 34 current snap is 5 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: finishing write tid 3 to smithi19530025-1 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: finishing write tid 4 to smithi19530025-1 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: oid 1 updating version 0 to 9 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: oid 1 updating version 9 to 10 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: oid 1 version 10 is already newer than 8 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 1 v 10 (ObjNum 56 snap 5 seq_num 56) dirty exists 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:78: left oid 1 (ObjNum 56 snap 5 seq_num 56) 2024-09-16T02:14:32.325 INFO:tasks.rados.rados.0.smithi195.stdout:84: expect (ObjNum 18 snap 0 seq_num 18) 2024-09-16T02:14:32.326 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 268864 2024-09-16T02:14:32.331 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 218101 2024-09-16T02:14:32.333 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 294466 2024-09-16T02:14:32.333 INFO:tasks.rados.rados.0.smithi195.stdout:81: finishing write tid 1 to smithi19530025-44 2024-09-16T02:14:32.333 INFO:tasks.rados.rados.0.smithi195.stdout:83: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-16T02:14:32.334 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:32.343 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 102090 2024-09-16T02:14:32.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T02:14:32.347 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:32.350 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:32.344+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: finishing write tid 2 to smithi19530025-44 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: finishing write tid 3 to smithi19530025-44 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: finishing write tid 4 to smithi19530025-44 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: oid 44 updating version 0 to 23 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: oid 44 updating version 23 to 24 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: oid 44 updating version 24 to 25 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 25 (ObjNum 57 snap 5 seq_num 57) dirty exists 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:81: left oid 44 (ObjNum 57 snap 5 seq_num 57) 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:80: finishing copy_from racing read to smithi19530025-26 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:80: finishing copy_from to smithi19530025-26 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 26 v 11 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T02:14:32.354 INFO:tasks.rados.rados.0.smithi195.stdout:90: seq_num 58 ranges {557056=606208,1163264=376832} 2024-09-16T02:14:32.359 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:32.359 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:32.372 INFO:tasks.rados.rados.0.smithi195.stdout:90: writing smithi19530025-34 from 557056 to 1163264 tid 1 2024-09-16T02:14:32.376 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:32.376 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:32.378 INFO:tasks.rados.rados.0.smithi195.stdout:90: writing smithi19530025-34 from 1163264 to 1540096 tid 2 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:78: done (9 left) 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:80: done (8 left) 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:81: done (7 left) 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:83: done (6 left) 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:84: done (5 left) 2024-09-16T02:14:32.379 INFO:tasks.rados.rados.0.smithi195.stdout:91: snap_remove snap 1 2024-09-16T02:14:32.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:32.388 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:32.391 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:14:32.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:32.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:32.438 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:32.438 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:32.451 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:32.446+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:32.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:32.459 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:32.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:32.485 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:32.525 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:32.525 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:32.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:32.548 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:32.552 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:32.546+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:32.561 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:32.562 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:32.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:32.569 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:32.576 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:14:32.596 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:32.596 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:32.624 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:32.624 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:32.647 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:32.648 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:32.653 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:32.652+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:32.675 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:32.675 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:32.699 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:32.699 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:32.699 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:32.699 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:32.705 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:32.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:32.706+0000 7fd3f0cb8640 0 -- 172.21.15.3:0/3369648265 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fd3ec2225e0 msgr2=0x7fd3ec224a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:32.714 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:14:32.744 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:32.744 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:32.754 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:32.750+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:32.775 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:32.775 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.278547+0000", 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5594792670000004, 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036513930000000002, 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.842 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278547+0000", 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278547+0000", 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.843 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278547+0000", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3700000000000001e-07 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278612+0000", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4616000000000006e-05 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278824+0000", 2024-09-16T02:14:32.844 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021251700000000001 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278844+0000", 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9820999999999999e-05 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278986+0000", 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014221500000000001 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.845 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279006+0000", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9834999999999999e-05 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.282118+0000", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031118700000000001 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.282198+0000", 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0081999999999997e-05 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.846 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.3s2 34/32 ECSubWrite(tid=112, reqid=client.4598.0:1, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.281838+0000", 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5561879980000004, 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034132559999999999, 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.847 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281838+0000", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281838+0000", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281839+0000", 2024-09-16T02:14:32.848 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5900000000000002e-07 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281891+0000", 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2503000000000002e-05 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281952+0000", 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0791999999999997e-05 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.849 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281977+0000", 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4916e-05 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.282158+0000", 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018103699999999999 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.282172+0000", 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4022000000000001e-05 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.850 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285205+0000", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030329129999999999 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285251+0000", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6613999999999998e-05 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:8, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.283234+0000", 2024-09-16T02:14:32.851 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5547922399999994, 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032103769999999999, 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283234+0000", 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.852 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283234+0000", 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283235+0000", 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.274e-06 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283265+0000", 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9539e-05 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.853 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283339+0000", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4017000000000006e-05 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283354+0000", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4868999999999999e-05 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283533+0000", 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000179085 2024-09-16T02:14:32.854 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5804e-05 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.286405+0000", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002856538 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.286444+0000", 2024-09-16T02:14:32.855 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9251000000000001e-05 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:31 3.1s0 3:89ce0a26:::smithi19530025-11:head [append 0~180224,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5473701819999999, 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022356674999999999, 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.856 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 31 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:32.857 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3159999999999999e-06 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290984+0000", 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032549899999999998 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.858 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290996+0000", 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2635e-05 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291004+0000", 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5440000000000001e-06 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.859 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291070+0000", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6177e-05 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291178+0000", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107766 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295206+0000", 2024-09-16T02:14:32.860 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040278029999999999 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299290+0000", 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040844139999999998 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312901+0000", 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013611165999999999 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.861 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313013+0000", 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011135499999999999 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:32 3.1s0 3:89ce0a26:::smithi19530025-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5469948510000009, 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022289759999999999, 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.862 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 32 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.863 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291032+0000", 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2600000000000001e-07 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291036+0000", 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2760000000000002e-06 2024-09-16T02:14:32.864 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291044+0000", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7190000000000001e-06 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291050+0000", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5779999999999998e-06 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295781+0000", 2024-09-16T02:14:32.865 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047316909999999997 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295883+0000", 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010134 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296411+0000", 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052795499999999998 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.866 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299292+0000", 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028809719999999999 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313180+0000", 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013887932 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313321+0000", 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141371 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.867 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:33 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5469626269999992, 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024514080000000001, 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 33 2024-09-16T02:14:32.868 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.869 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291064+0000", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291067+0000", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7199999999999998e-06 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291074+0000", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4680000000000003e-06 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.870 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291080+0000", 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.964e-06 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055391950000000002 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0838000000000001e-05 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.871 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313405+0000", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016704404999999999 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313545+0000", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014004999999999999 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315578+0000", 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020328199999999999 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.872 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:37 3.1s0 3:81c6c0a8:::smithi19530025-13:head [append 0~212992,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5469341800000009, 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023064575, 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.873 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 37 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.874 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291093+0000", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4399999999999999e-07 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291317+0000", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022428500000000001 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291335+0000", 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7980999999999999e-05 2024-09-16T02:14:32.875 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291347+0000", 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1708e-05 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296736+0000", 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053889660000000002 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.876 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296859+0000", 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123786 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297520+0000", 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066093900000000004 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302005+0000", 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044842199999999997 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.877 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314113+0000", 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012108659000000001 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314156+0000", 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3186999999999998e-05 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.878 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:38 3.1s0 3:81c6c0a8:::smithi19530025-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5466646950000005, 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023044124999999999, 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 38 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.879 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291362+0000", 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0100000000000001e-06 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.880 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291368+0000", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6879999999999996e-06 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291381+0000", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3369e-05 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291404+0000", 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2164e-05 2024-09-16T02:14:32.881 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298518+0000", 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007114029 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298598+0000", 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0813e-05 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.882 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299017+0000", 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041827500000000002 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302007+0000", 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029904630000000001 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314313+0000", 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012305768999999999 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.883 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314405+0000", 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2545000000000002e-05 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:39 3.1s0 3:81c6c0a8:::smithi19530025-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5465927619999995, 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024374809000000001, 2024-09-16T02:14:32.884 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 39 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.885 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291434+0000", 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.886 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291440+0000", 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4960000000000004e-06 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291456+0000", 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025e-05 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291468+0000", 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-16T02:14:32.887 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299421+0000", 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079526309999999999 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299510+0000", 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9506000000000007e-05 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.888 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314495+0000", 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014984548 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314572+0000", 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6854999999999995e-05 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315808+0000", 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012362639999999999 2024-09-16T02:14:32.889 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:40 3.1s0 3:830a71a8:::smithi19530025-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5465411699999994, 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024497693000000001, 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.890 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 40 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.891 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291486+0000", 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8999999999999995e-07 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.892 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291773+0000", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028741999999999998 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291795+0000", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.181e-05 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291807+0000", 2024-09-16T02:14:32.893 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1687e-05 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299547+0000", 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077406699999999998 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299647+0000", 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9978999999999999e-05 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.894 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.300445+0000", 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079766400000000001 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302061+0000", 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016158539999999999 2024-09-16T02:14:32.895 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315913+0000", 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013852573999999999 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315983+0000", 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9145000000000003e-05 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.896 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:41 3.1s0 3:830a71a8:::smithi19530025-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5461938360000005, 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024477925000000001, 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.897 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.898 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291833+0000", 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291839+0000", 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-16T02:14:32.899 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291855+0000", 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291865+0000", 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0037999999999999e-05 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.900 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301194+0000", 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093284930000000002 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301281+0000", 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6972999999999996e-05 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.901 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301794+0000", 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051343099999999996 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.304892+0000", 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030976950000000001 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316223+0000", 2024-09-16T02:14:32.902 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011331071999999999 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316310+0000", 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.742e-05 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.903 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:42 3.1s0 3:830a71a8:::smithi19530025-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5461476439999995, 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031481841000000003, 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.904 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.905 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291880+0000", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2139999999999999e-06 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291883+0000", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291902+0000", 2024-09-16T02:14:32.906 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8881000000000001e-05 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291912+0000", 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9680000000000001e-06 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.907 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302284+0000", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371521999999999 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302334+0000", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0047999999999999e-05 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318985+0000", 2024-09-16T02:14:32.908 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016650888999999999 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.319121+0000", 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136336 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323360+0000", 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042391670000000003 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.909 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:64 3.1s0 3:8151366d:::smithi19530025-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5196830420000005, 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014450049, 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.910 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 64 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.911 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318345+0000", 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.75e-06 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318434+0000", 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9405000000000002e-05 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.912 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318448+0000", 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4042999999999999e-05 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318456+0000", 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9289999999999998e-06 2024-09-16T02:14:32.913 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.320462+0000", 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002006271 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.320696+0000", 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000233231 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.914 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.321663+0000", 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096740900000000002 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323488+0000", 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001825089 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332528+0000", 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090399920000000002 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.915 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332793+0000", 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026493000000000003 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:65 3.1s0 3:8151366d:::smithi19530025-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5195519480000002, 2024-09-16T02:14:32.916 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014507262999999999, 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 65 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:32.917 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318475+0000", 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3600000000000003e-07 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.918 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318480+0000", 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3000000000000001e-06 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318488+0000", 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4479999999999997e-06 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.919 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318493+0000", 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7200000000000003e-06 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322383+0000", 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038899780000000001 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322473+0000", 2024-09-16T02:14:32.920 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0278000000000005e-05 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322933+0000", 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045975500000000001 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323635+0000", 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00070165699999999995 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.921 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332907+0000", 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092716980000000001 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332981+0000", 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4793e-05 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:66 3.1s0 3:8151366d:::smithi19530025-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.922 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5195230249999998, 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016162184999999999, 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 66 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.923 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.924 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318504+0000", 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3799999999999997e-07 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318507+0000", 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7469999999999999e-06 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.925 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318515+0000", 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2250000000000003e-06 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318521+0000", 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3269999999999997e-06 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323407+0000", 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004885777 2024-09-16T02:14:32.926 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323480+0000", 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3491000000000005e-05 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.333374+0000", 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098940399999999998 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.333475+0000", 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010128099999999999 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334665+0000", 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011897590000000001 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.928 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.3s2 35/32 ECSubWrite(tid=125, reqid=client.4598.0:79, at_version=35'3, trim_to=0'0, roll_forward_to=34'2))", 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.336623+0000", 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.5014028029999995, 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035869840000000001, 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:32.929 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336623+0000", 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336623+0000", 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336624+0000", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336927+0000", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030282799999999999 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336964+0000", 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7301000000000001e-05 2024-09-16T02:14:32.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.336974+0000", 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9520000000000006e-06 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337044+0000", 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9411000000000003e-05 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.932 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337053+0000", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.081e-06 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.340169+0000", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031163369999999998 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.340210+0000", 2024-09-16T02:14:32.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1297000000000001e-05 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:121 3.1s0 3:830b1788:::smithi19530025-41:head [append 0~688128,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.387893+0000", 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.4501328050000009, 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061455429999999998, 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.934 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 121 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.387893+0000", 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.387893+0000", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.387895+0000", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1749999999999999e-06 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388305+0000", 2024-09-16T02:14:32.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00040926899999999997 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388321+0000", 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.628e-05 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388330+0000", 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9290000000000004e-06 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388416+0000", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5886999999999999e-05 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388549+0000", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013302 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.389437+0000", 2024-09-16T02:14:32.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088859400000000004 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.391230+0000", 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001792472 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.393968+0000", 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002737869 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.394039+0000", 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1048000000000001e-05 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:122 3.1s0 3:830b1788:::smithi19530025-41:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.940 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.388355+0000", 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.4496711229999999, 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.008584576, 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 122 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388355+0000", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388355+0000", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388356+0000", 2024-09-16T02:14:32.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5300000000000003e-07 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388360+0000", 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9480000000000001e-06 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388371+0000", 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1368e-05 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388378+0000", 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8199999999999999e-06 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.390153+0000", 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001775125 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.390290+0000", 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136787 2024-09-16T02:14:32.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.390763+0000", 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047294500000000001 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.391313+0000", 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00055059 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.396871+0000", 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055577639999999998 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.396940+0000", 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8476e-05 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:123 3.1s0 3:830b1788:::smithi19530025-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.388388+0000", 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.4496381469999999, 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0097600780000000002, 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 123 2024-09-16T02:14:32.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388388+0000", 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388388+0000", 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388389+0000", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8999999999999996e-07 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388391+0000", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4279999999999999e-06 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388399+0000", 2024-09-16T02:14:32.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8709999999999997e-06 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.388404+0000", 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.49e-06 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.391009+0000", 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002604229 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.950 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.391109+0000", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100259 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.397248+0000", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006139327 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.397340+0000", 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2238999999999996e-05 2024-09-16T02:14:32.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.398148+0000", 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00080764499999999996 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:130 3.1s0 3:8203d376:::smithi19530025-44:head [append 0~737280,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.402911+0000", 2024-09-16T02:14:32.952 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.435115283, 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039866830000000004, 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 130 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.402911+0000", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.402911+0000", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.402913+0000", 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0370000000000001e-06 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403105+0000", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019177899999999999 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403124+0000", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9477e-05 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403134+0000", 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0035e-05 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403198+0000", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3805000000000003e-05 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403306+0000", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107965 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.403974+0000", 2024-09-16T02:14:32.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066825100000000002 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.405363+0000", 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001388717 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.406814+0000", 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001451162 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:32.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.406897+0000", 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3455000000000006e-05 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:32.958 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:32.960 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:14:32.960 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:32.956+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:33.061 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:33.056+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:33.086 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:33.086 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:33.086 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:33.086 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:33.100 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:14:33.162 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:33.162+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:33.263 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:33.258+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:33.268 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:33.268 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:33.268 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:33.268 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:33.268 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:33.283 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:14:33.328 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T02:14:33.333 INFO:tasks.rados.rados.0.smithi195.stdout:91: done (5 left) 2024-09-16T02:14:33.334 INFO:tasks.rados.rados.0.smithi195.stdout:92: copy_from oid 40 from oid 35 current snap is 5 2024-09-16T02:14:33.334 INFO:tasks.rados.rados.0.smithi195.stdout:82: finishing copy_from racing read to smithi19530025-32 2024-09-16T02:14:33.334 INFO:tasks.rados.rados.0.smithi195.stdout:86: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-16T02:14:33.339 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 312098 2024-09-16T02:14:33.350 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:33.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:14:33.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T02:14:33.364 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:33.364+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:33.374 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 539826 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:82: finishing copy_from to smithi19530025-32 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 32 v 2 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:87: finishing rollback tid 0 to smithi19530025-49 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:87: finishing rollback tid 1 to smithi19530025-49 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:82: done (5 left) 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:86: done (4 left) 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:93: setattr oid 11 current snap is 5 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:87: finishing rollback tid 2 to smithi19530025-49 2024-09-16T02:14:33.376 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 49 v 20 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-16T02:14:33.377 INFO:tasks.rados.rados.0.smithi195.stdout:89: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-16T02:14:33.377 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:33.391 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:33.412 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 555680 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: finishing write tid 1 to smithi19530025-34 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: finishing write tid 2 to smithi19530025-34 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: finishing write tid 3 to smithi19530025-34 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: finishing write tid 4 to smithi19530025-34 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: oid 34 updating version 0 to 22 2024-09-16T02:14:33.417 INFO:tasks.rados.rados.0.smithi195.stdout:90: oid 34 updating version 22 to 23 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/32 ECSubWrite(tid=0, reqid=client.4598.0:7, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.278627+0000", 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1367281390000006, 2024-09-16T02:14:33.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004928338, 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278627+0000", 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278627+0000", 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278627+0000", 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7399999999999999e-07 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278704+0000", 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6835000000000006e-05 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278857+0000", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000153225 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.278868+0000", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0665999999999999e-05 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279180+0000", 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031241899999999998 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279203+0000", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2977999999999999e-05 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283519+0000", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004316205 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283555+0000", 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5735999999999998e-05 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.422 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1361197409999999, 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.423 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:14:33.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.428 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1359269659999995, 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:14:33.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:14:33.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:33.433 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:33.434 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1358689230000003, 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:33.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:14:33.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.438 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:14:33.439 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/32 ECSubWrite(tid=2, reqid=client.4598.0:8, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.280269+0000", 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1350855929999994, 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039621810000000004, 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.440 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280269+0000", 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280269+0000", 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.441 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280270+0000", 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7300000000000005e-07 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280288+0000", 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7952e-05 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280333+0000", 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4929000000000003e-05 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.442 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280341+0000", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5949999999999999e-06 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280406+0000", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4150000000000001e-05 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280415+0000", 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5950000000000005e-06 2024-09-16T02:14:33.443 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284201+0000", 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037854469999999999 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284231+0000", 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.074e-05 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 34/32 ECSubWrite(tid=1, reqid=client.4598.0:4, at_version=34'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.280908+0000", 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1344469969999995, 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034001809999999999, 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280908+0000", 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280908+0000", 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280908+0000", 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5400000000000002e-07 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.280952+0000", 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3789999999999999e-05 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281005+0000", 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2895000000000001e-05 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281016+0000", 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0854e-05 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281151+0000", 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013572399999999999 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281163+0000", 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1934e-05 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284278+0000", 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031145510000000001 2024-09-16T02:14:33.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284308+0000", 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9978999999999999e-05 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:33.450 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.298867+0000", 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1164878100000006, 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033803520000000001, 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298867+0000", 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298867+0000", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298868+0000", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2720000000000001e-06 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298888+0000", 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0194e-05 2024-09-16T02:14:33.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298954+0000", 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5242999999999999e-05 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298967+0000", 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2951e-05 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302229+0000", 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032621379999999999 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302247+0000", 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8553999999999999e-05 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 35/32 ECSubWrite(tid=4, reqid=client.4598.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.298985+0000", 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1163702170000001, 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003479879, 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298985+0000", 2024-09-16T02:14:33.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298985+0000", 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298985+0000", 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1800000000000005e-07 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298996+0000", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0220999999999999e-05 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299080+0000", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4505999999999994e-05 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299088+0000", 2024-09-16T02:14:33.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4030000000000007e-06 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299439+0000", 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035009899999999998 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9190000000000005e-06 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302408+0000", 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029604969999999999 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6416000000000003e-05 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 35/32 ECSubWrite(tid=4, reqid=client.4598.0:44, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.299258+0000", 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1160968750000002, 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031296219999999999, 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299258+0000", 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299258+0000", 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299259+0000", 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1379999999999999e-06 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299292+0000", 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2839999999999997e-05 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299348+0000", 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5742000000000003e-05 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299363+0000", 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5515e-05 2024-09-16T02:14:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299454+0000", 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0810000000000006e-05 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299463+0000", 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7520000000000002e-06 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302283+0000", 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028208389999999999 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302387+0000", 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103986 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.464 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:52 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1070013589999999, 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010472922000000001, 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 52 2024-09-16T02:14:33.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.466 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308356+0000", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3439999999999999e-06 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308602+0000", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024608900000000001 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308617+0000", 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5695000000000001e-05 2024-09-16T02:14:33.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308627+0000", 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0759999999999992e-06 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308714+0000", 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7399999999999997e-05 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308908+0000", 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019356000000000001 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.310117+0000", 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001209679 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312215+0000", 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020981490000000001 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318705+0000", 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064893049999999999 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318826+0000", 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121625 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1067119929999993, 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.471 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:14:33.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1066589340000004, 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.476 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:90: oid 34 updating version 23 to 24 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 24 (ObjNum 58 snap 5 seq_num 58) dirty exists 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:90: left oid 34 (ObjNum 58 snap 5 seq_num 58) 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:92: finishing copy_from to smithi19530025-40 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:92: got expected ENOENT (src dne) 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:92: finishing copy_from racing read to smithi19530025-40 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:87: done (4 left) 2024-09-16T02:14:33.479 INFO:tasks.rados.rados.0.smithi195.stdout:89: done (3 left) 2024-09-16T02:14:33.480 INFO:tasks.rados.rados.0.smithi195.stdout:90: done (2 left) 2024-09-16T02:14:33.480 INFO:tasks.rados.rados.0.smithi195.stdout:92: done (1 left) 2024-09-16T02:14:33.480 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 48 current snap is 5 2024-09-16T02:14:33.480 INFO:tasks.rados.rados.0.smithi195.stdout:94: seq_num 59 ranges {802816=622592,1425408=376832} 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:14:33.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 35/32 ECSubWriteReply(tid=2, last_complete=35'2, committed=1, applied=1))", 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308706+0000", 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.1066484659999993, 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035109910000000002, 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.483 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308706+0000", 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308706+0000", 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3099999999999997e-07 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308715+0000", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5179999999999998e-06 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308751+0000", 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6748000000000001e-05 2024-09-16T02:14:33.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308759+0000", 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7719999999999994e-06 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312204+0000", 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034448349999999998 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312217+0000", 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3387000000000001e-05 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.0s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:47, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.315376+0000", 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.0999789690000004, 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003224696, 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.487 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315376+0000", 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315376+0000", 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315377+0000", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-07 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315387+0000", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0835999999999999e-05 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315424+0000", 2024-09-16T02:14:33.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7008000000000002e-05 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315433+0000", 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6950000000000006e-06 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315497+0000", 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3530999999999999e-05 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315528+0000", 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1198999999999997e-05 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318560+0000", 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030316969999999999 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318600+0000", 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0912999999999999e-05 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=8, reqid=client.4598.0:23, at_version=35'6, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.334787+0000", 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.0805675350000001, 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0045909119999999999, 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334787+0000", 2024-09-16T02:14:33.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334787+0000", 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334788+0000", 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334807+0000", 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9194999999999998e-05 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338087+0000", 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032793480000000001 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338099+0000", 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2326e-05 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338234+0000", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013493500000000001 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338246+0000", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2615e-05 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.339335+0000", 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001088072 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.339378+0000", 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3594999999999998e-05 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=9, reqid=client.4598.0:73, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.0774095989999992, 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075239130000000001, 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.496 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337946+0000", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338051+0000", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104225 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343980+0000", 2024-09-16T02:14:33.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059297350000000002 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343990+0000", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4040000000000001e-06 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.344047+0000", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7324000000000002e-05 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.344058+0000", 2024-09-16T02:14:33.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0813e-05 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.345433+0000", 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013752059999999999 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.345469+0000", 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5880000000000002e-05 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.500 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.0714774709999997, 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:14:33.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:97, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.363392+0000", 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.0519626520000003, 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038045800000000001, 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.363392+0000", 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.363392+0000", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.363393+0000", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4099999999999997e-07 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.364601+0000", 2024-09-16T02:14:33.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001207631 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.364675+0000", 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4567999999999995e-05 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.364682+0000", 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6699999999999997e-06 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.364899+0000", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000217109 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.364908+0000", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3889999999999994e-06 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.367158+0000", 2024-09-16T02:14:33.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002249334 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.367197+0000", 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9038000000000001e-05 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 44/32 ECSubWrite(tid=29, reqid=osd.5.0:29, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334365+0000", 2024-09-16T02:14:33.510 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.080989518999999996, 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003575571, 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334365+0000", 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334365+0000", 2024-09-16T02:14:33.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334366+0000", 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8299999999999997e-07 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334408+0000", 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2209000000000002e-05 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334488+0000", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0581999999999995e-05 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334506+0000", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7088999999999999e-05 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334671+0000", 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016528499999999999 2024-09-16T02:14:33.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334690+0000", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8751999999999999e-05 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.337905+0000", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032155920000000002 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.337941+0000", 2024-09-16T02:14:33.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5578999999999999e-05 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s4 44/32 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.346909+0000", 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.068445650999999996, 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003069656, 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:33.515 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.346909+0000", 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.346909+0000", 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.346909+0000", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.97e-07 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.346932+0000", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3213e-05 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.347038+0000", 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105998 2024-09-16T02:14:33.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.347047+0000", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6580000000000008e-06 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.347113+0000", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5708000000000002e-05 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.347122+0000", 2024-09-16T02:14:33.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4080000000000004e-06 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.349938+0000", 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002815667 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.349979+0000", 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0806999999999997e-05 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.519 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.520 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.520 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:33.520 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:33.520 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:33.520 INFO:tasks.rados.rados.0.smithi195.stdout:94: writing smithi19530025-48 from 802816 to 1425408 tid 1 2024-09-16T02:14:33.520 INFO:tasks.rados.rados.0.smithi195.stdout:94: writing smithi19530025-48 from 1425408 to 1802240 tid 2 2024-09-16T02:14:33.521 INFO:tasks.rados.rados.0.smithi195.stdout:95: snap_remove snap 0 2024-09-16T02:14:33.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:33.525 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:33.518+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:33.626 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:33.626+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:33.666 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:33.666 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:33.666 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:33.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:33.727 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:33.724+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:33.807 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:33.808 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:33.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:33.827 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:33.824+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:33.928 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:33.926+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:33.950 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:33.950 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:33.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:33.950 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:33.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'20)", 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.367971423, 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055114170000000002, 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578972+0000", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0299999999999998e-07 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578977+0000", 2024-09-16T02:14:33.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6269999999999997e-06 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579012+0000", 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4903000000000001e-05 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579023+0000", 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0949e-05 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579077+0000", 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3783000000000003e-05 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579092+0000", 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5566999999999999e-05 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584412+0000", 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053199859999999996 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584482+0000", 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9899000000000003e-05 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:44 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'26, mlcod=24'20)", 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.579999+0000", 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.366943256999999, 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045432270000000004, 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.955 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579999+0000", 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579999+0000", 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580000+0000", 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4499999999999997e-07 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580103+0000", 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103664 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580140+0000", 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6677000000000001e-05 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580148+0000", 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4830000000000002e-06 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580221+0000", 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3181000000000003e-05 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.580256+0000", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4768000000000003e-05 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584486+0000", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042308160000000001 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584542+0000", 2024-09-16T02:14:33.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5992999999999997e-05 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:70 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'48, mlcod=24'42)", 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.682161+0000", 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.264781545000002, 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045748079999999997, 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682161+0000", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682161+0000", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682163+0000", 2024-09-16T02:14:33.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7260000000000001e-06 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682174+0000", 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1287000000000001e-05 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682246+0000", 2024-09-16T02:14:33.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2496000000000005e-05 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682258+0000", 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1552999999999999e-05 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682771+0000", 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00051339599999999995 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.682836+0000", 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4503000000000002e-05 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686675+0000", 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038395069999999998 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686736+0000", 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0340000000000002e-05 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.964 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:74 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'52, mlcod=24'45)", 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.261752176999998, 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050466110000000003, 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685192+0000", 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5400000000000001e-06 2024-09-16T02:14:33.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685202+0000", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0287e-05 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685261+0000", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8758999999999998e-05 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685273+0000", 2024-09-16T02:14:33.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.261e-05 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686230+0000", 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095673999999999998 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686263+0000", 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3319999999999999e-05 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.690175+0000", 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039115419999999996 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.690237+0000", 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1812999999999999e-05 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'46)", 2024-09-16T02:14:33.969 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.260701758, 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060039550000000001, 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686242+0000", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.564e-06 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686348+0000", 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105983 2024-09-16T02:14:33.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686445+0000", 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6599000000000001e-05 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686462+0000", 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7557000000000001e-05 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686971+0000", 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050895900000000002 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687026+0000", 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4799999999999997e-05 2024-09-16T02:14:33.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.692166+0000", 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051400029999999998 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.692245+0000", 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8490000000000002e-05 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:76 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'54, mlcod=24'46)", 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.259424133, 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007242999, 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687520+0000", 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.796e-06 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687531+0000", 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1420000000000001e-05 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687607+0000", 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5166999999999994e-05 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687620+0000", 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3873000000000001e-05 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689759+0000", 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021384210000000002 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689788+0000", 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8782999999999999e-05 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694680+0000", 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004892123 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694761+0000", 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1415999999999994e-05 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:78 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'56, mlcod=24'53)", 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.697973+0000", 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.248968940000001, 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040453219999999996, 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.697973+0000", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.697973+0000", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.697975+0000", 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9039999999999999e-06 2024-09-16T02:14:33.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.698093+0000", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118139 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.698187+0000", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3479000000000004e-05 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.698207+0000", 2024-09-16T02:14:33.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.986e-05 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.698355+0000", 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014790199999999999 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.698431+0000", 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6742999999999994e-05 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.701935+0000", 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035040259999999999 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.702019+0000", 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3269000000000005e-05 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:95 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'69, mlcod=24'65)", 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:33.984 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.14170876, 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057000499999999999, 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:33.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805235+0000", 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9999999999999995e-07 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805244+0000", 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1260000000000004e-06 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805294+0000", 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.02e-05 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805306+0000", 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.217e-05 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805406+0000", 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9481999999999994e-05 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805441+0000", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5247000000000003e-05 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810846+0000", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054056449999999997 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810934+0000", 2024-09-16T02:14:33.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7180000000000002e-05 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:96 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'70, mlcod=24'65)", 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.140574812000001, 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063466429999999999, 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.989 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806369+0000", 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.297e-06 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806477+0000", 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107983 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806532+0000", 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5075999999999999e-05 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806544+0000", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2405999999999999e-05 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806623+0000", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8689999999999994e-05 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806656+0000", 2024-09-16T02:14:33.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3102999999999998e-05 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812648+0000", 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059919650000000001 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812714+0000", 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6123000000000004e-05 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:97 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'71, mlcod=24'66)", 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.138434683, 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056618140000000003, 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808509+0000", 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.823e-06 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808521+0000", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1759999999999999e-05 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808597+0000", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5833000000000004e-05 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808615+0000", 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8292999999999999e-05 2024-09-16T02:14:33.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810373+0000", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001757315 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810436+0000", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2846000000000001e-05 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814082+0000", 2024-09-16T02:14:33.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036462980000000001 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814169+0000", 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7645999999999994e-05 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:98 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:33.998 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.136996022999998, 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005045051, 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:33.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809948+0000", 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.46e-06 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809958+0000", 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0596e-05 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810018+0000", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9833000000000001e-05 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810029+0000", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029999999999999e-05 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.811011+0000", 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098175699999999999 2024-09-16T02:14:34.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.811066+0000", 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4765999999999997e-05 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814911+0000", 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003844737 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814991+0000", 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0871999999999994e-05 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:104 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'78, mlcod=24'73)", 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.126667629, 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050841580000000001, 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820276+0000", 2024-09-16T02:14:34.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820326+0000", 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0062999999999998e-05 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820380+0000", 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4562999999999999e-05 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820390+0000", 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8970000000000003e-06 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820465+0000", 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4332999999999995e-05 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820494+0000", 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9272999999999999e-05 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.825267+0000", 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047728459999999999 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.825359+0000", 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2152999999999997e-05 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.007 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:114 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'84, mlcod=24'80)", 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.909341+0000", 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.037601142, 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043560769999999999, 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.909341+0000", 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.909341+0000", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.909343+0000", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8929999999999999e-06 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.909395+0000", 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2200000000000002e-05 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.910870+0000", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014745940000000001 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.910886+0000", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5899000000000001e-05 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911018+0000", 2024-09-16T02:14:34.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013233599999999999 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911052+0000", 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3559999999999997e-05 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913611+0000", 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025592219999999999 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913697+0000", 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6372999999999994e-05 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:116 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'86, mlcod=24'80)", 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.035075916, 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005826726, 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.012 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911868+0000", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911878+0000", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0699999999999999e-05 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913929+0000", 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020504820000000002 2024-09-16T02:14:34.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913942+0000", 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.327e-05 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.914088+0000", 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000145642 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.914122+0000", 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4597000000000001e-05 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.917632+0000", 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035092779999999998 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.917693+0000", 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1416999999999998e-05 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.016 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:117 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'87, mlcod=24'82)", 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.913839+0000", 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.033103402999998, 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042491400000000002, 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913839+0000", 2024-09-16T02:14:34.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913839+0000", 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913840+0000", 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.39e-06 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913893+0000", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3149999999999998e-05 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.915179+0000", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001285904 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.915199+0000", 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9296e-05 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.915326+0000", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127503 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.915365+0000", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8896000000000002e-05 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.917998+0000", 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002633008 2024-09-16T02:14:34.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.918088+0000", 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9993000000000004e-05 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:121 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'91, mlcod=24'84)", 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.919241+0000", 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.027701190999998, 2024-09-16T02:14:34.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048031790000000003, 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.919241+0000", 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.919241+0000", 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.919242+0000", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.207e-06 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.919253+0000", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0726e-05 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921187+0000", 2024-09-16T02:14:34.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019336259999999999 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921202+0000", 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5767e-05 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921358+0000", 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015525200000000001 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921418+0000", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0449999999999999e-05 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.923938+0000", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025196390000000002 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924044+0000", 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106512 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.025 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'87)", 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.025820593999999, 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055942539999999999, 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921125+0000", 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7999999999999999e-06 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921139+0000", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4912999999999999e-05 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922440+0000", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013005670000000001 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922453+0000", 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2476999999999999e-05 2024-09-16T02:14:34.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922551+0000", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8579999999999995e-05 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922584+0000", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3172e-05 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.926588+0000", 2024-09-16T02:14:34.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040037329999999998 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.926716+0000", 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012801199999999999 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'87)", 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:34.030 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.024627710000001, 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069325319999999999, 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922316+0000", 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4640000000000001e-06 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922381+0000", 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4969999999999996e-05 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924466+0000", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002085075 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924479+0000", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3033e-05 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924623+0000", 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-09-16T02:14:34.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924687+0000", 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3737000000000003e-05 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929168+0000", 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044810020000000004 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929247+0000", 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9472000000000001e-05 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'89)", 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.022521999999999, 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33740482100000002, 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924422+0000", 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924431+0000", 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8769999999999999e-06 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260013+0000", 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33558222100000001 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260031+0000", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8063e-05 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260228+0000", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000196782 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260273+0000", 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877000000000001e-05 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261745+0000", 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001471916 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261825+0000", 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9791999999999998e-05 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 44/32 ECSubWrite(tid=43, reqid=osd.11.0:43, at_version=44'35, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.349325+0000", 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.59761763700000003, 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044209280000000002, 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349325+0000", 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349325+0000", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349326+0000", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0400000000000005e-07 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349344+0000", 2024-09-16T02:14:34.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8025000000000001e-05 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349381+0000", 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7367000000000003e-05 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349389+0000", 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5529999999999993e-06 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349476+0000", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7000000000000001e-05 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.349503+0000", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7018000000000001e-05 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353688+0000", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041843089999999998 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353746+0000", 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7852000000000001e-05 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.044 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:34.046 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:34.038+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:34.147 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:34.143+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:34.216 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:34.217 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:34.217 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:34.217 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:34.248 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:34.244+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:34.335 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:14:34.336 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 36 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: finishing write tid 1 to smithi19530025-48 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: finishing write tid 2 to smithi19530025-48 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: finishing write tid 3 to smithi19530025-48 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: finishing write tid 4 to smithi19530025-48 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: oid 48 updating version 0 to 38 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: oid 48 updating version 38 to 40 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: oid 48 version 40 is already newer than 39 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 40 (ObjNum 59 snap 5 seq_num 59) dirty exists 2024-09-16T02:14:34.344 INFO:tasks.rados.rados.0.smithi195.stdout:94: left oid 48 (ObjNum 59 snap 5 seq_num 59) 2024-09-16T02:14:34.345 INFO:tasks.rados.rados.0.smithi195.stdout:93: done (2 left) 2024-09-16T02:14:34.345 INFO:tasks.rados.rados.0.smithi195.stdout:94: done (1 left) 2024-09-16T02:14:34.345 INFO:tasks.rados.rados.0.smithi195.stdout:95: done (0 left) 2024-09-16T02:14:34.345 INFO:tasks.rados.rados.0.smithi195.stdout:96: copy_from oid 11 from oid 33 current snap is 5 2024-09-16T02:14:34.345 INFO:tasks.rados.rados.0.smithi195.stdout:97: delete oid 16 current snap is 5 2024-09-16T02:14:34.349 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:34.342+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:34.357 INFO:tasks.rados.rados.0.smithi195.stdout:97: done (1 left) 2024-09-16T02:14:34.357 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 8 current snap is 5 2024-09-16T02:14:34.357 INFO:tasks.rados.rados.0.smithi195.stdout:98: seq_num 61 ranges {131072=655360,786432=376832} 2024-09-16T02:14:34.372 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T02:14:34.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:34.374 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:34.374 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:34.374 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.382 INFO:tasks.rados.rados.0.smithi195.stdout:98: writing smithi19530025-8 from 131072 to 786432 tid 1 2024-09-16T02:14:34.388 INFO:tasks.rados.rados.0.smithi195.stdout:98: writing smithi19530025-8 from 786432 to 1163264 tid 2 2024-09-16T02:14:34.388 INFO:tasks.rados.rados.0.smithi195.stdout:99: delete oid 45 current snap is 5 2024-09-16T02:14:34.388 INFO:tasks.rados.rados.0.smithi195.stdout:96: finishing copy_from racing read to smithi19530025-11 2024-09-16T02:14:34.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:34.391 INFO:tasks.rados.rados.0.smithi195.stdout:96: finishing copy_from to smithi19530025-11 2024-09-16T02:14:34.391 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 2 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T02:14:34.393 INFO:tasks.rados.rados.0.smithi195.stdout:96: done (2 left) 2024-09-16T02:14:34.393 INFO:tasks.rados.rados.0.smithi195.stdout:99: done (1 left) 2024-09-16T02:14:34.393 INFO:tasks.rados.rados.0.smithi195.stdout:100: snap_create 2024-09-16T02:14:34.394 INFO:tasks.rados.rados.0.smithi195.stdout:98: finishing write tid 1 to smithi19530025-8 2024-09-16T02:14:34.398 INFO:tasks.rados.rados.0.smithi195.stdout:98: finishing write tid 2 to smithi19530025-8 2024-09-16T02:14:34.450 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:34.448+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:34.551 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:34.550+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.2455527909999997, 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001906184, 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318699+0000", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3939999999999999e-06 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318830+0000", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013092799999999999 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318876+0000", 2024-09-16T02:14:34.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6335999999999998e-05 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318883+0000", 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1550000000000001e-06 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318984+0000", 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100424 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318995+0000", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0835e-05 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.320576+0000", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001581432 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.320604+0000", 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7679999999999999e-05 2024-09-16T02:14:34.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 38/32 ECSubWrite(tid=17, reqid=client.4598.0:193, at_version=38'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:27.323621+0000", 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.2406296880000003, 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001202226, 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323621+0000", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323621+0000", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323622+0000", 2024-09-16T02:14:34.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.362e-06 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323679+0000", 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7503999999999997e-05 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323729+0000", 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9833999999999998e-05 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323741+0000", 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1711e-05 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323811+0000", 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9499000000000007e-05 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.323820+0000", 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7909999999999999e-06 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.324738+0000", 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091773899999999999 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.324823+0000", 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4785999999999998e-05 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.576 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.8s3 38/32 ECSubRead(tid=21, to_read={3:165ce5f6:::smithi19530025-40:head=0,167936,0}, subchunks={3:165ce5f6:::smithi19530025-40:head=[0,1]}, attrs_to_read=))", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:27.339805+0000", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.2244450340000004, 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093989799999999999, 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339805+0000", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339805+0000", 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339807+0000", 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7549999999999999e-06 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339824+0000", 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6888000000000002e-05 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339871+0000", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7125e-05 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339883+0000", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1496e-05 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.339979+0000", 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.645e-05 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.340745+0000", 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00076618400000000005 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 44/32 ECSubWrite(tid=29, reqid=osd.5.0:29, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.333916+0000", 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.2303342100000001, 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017584650000000001, 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.580 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.333916+0000", 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.333916+0000", 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.333916+0000", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.53e-07 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.333953+0000", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.646e-05 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334031+0000", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8695999999999993e-05 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334045+0000", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3974999999999999e-05 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334212+0000", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016636099999999999 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334238+0000", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5913000000000001e-05 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.335630+0000", 2024-09-16T02:14:34.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001391883 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.335675+0000", 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4923999999999999e-05 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.334576+0000", 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.2296739640000001, 2024-09-16T02:14:34.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001654731, 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334576+0000", 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334576+0000", 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334577+0000", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3799999999999998e-07 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334647+0000", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0773999999999997e-05 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334674+0000", 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6316999999999999e-05 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334698+0000", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4131e-05 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334773+0000", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4745000000000003e-05 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.334783+0000", 2024-09-16T02:14:34.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0063e-05 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.336190+0000", 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001406972 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.336231+0000", 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1390999999999997e-05 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.343445+0000", 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.220805106, 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094000099999999999, 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343445+0000", 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343445+0000", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343446+0000", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8000000000000001e-07 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343469+0000", 2024-09-16T02:14:34.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3266000000000001e-05 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343522+0000", 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2979000000000002e-05 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343534+0000", 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1984000000000001e-05 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343606+0000", 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2292000000000005e-05 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.343613+0000", 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6789999999999999e-06 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.344346+0000", 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073365799999999999 2024-09-16T02:14:34.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.344385+0000", 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8763000000000001e-05 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 44/32 ECSubWrite(tid=189, reqid=osd.13.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:34.593 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.352961+0000", 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.2112892150000001, 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001843599, 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352961+0000", 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352961+0000", 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352962+0000", 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352970+0000", 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7609999999999996e-06 2024-09-16T02:14:34.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353076+0000", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105319 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353085+0000", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9479999999999997e-06 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353223+0000", 2024-09-16T02:14:34.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000138584 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.353250+0000", 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.709e-05 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.354765+0000", 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001515076 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.354805+0000", 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9310999999999997e-05 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 44/32 ECSubWrite(tid=23, reqid=osd.2.0:22, at_version=0'0, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.094791+0000", 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.46945973800000002, 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001618459, 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094791+0000", 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094791+0000", 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094792+0000", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9099999999999999e-06 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094817+0000", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4556999999999999e-05 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094901+0000", 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4386000000000001e-05 2024-09-16T02:14:34.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.094920+0000", 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.84e-05 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.095062+0000", 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014179199999999999 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.095080+0000", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8410000000000002e-05 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.096316+0000", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012356909999999999 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.096409+0000", 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3313000000000006e-05 2024-09-16T02:14:34.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=31, reqid=osd.5.0:31, at_version=45'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.341243+0000", 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.22300747500000001, 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00170732, 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341243+0000", 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341243+0000", 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341243+0000", 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6600000000000002e-07 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341277+0000", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3519999999999998e-05 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341357+0000", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0098000000000005e-05 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341373+0000", 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6047999999999999e-05 2024-09-16T02:14:34.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341458+0000", 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4679999999999996e-05 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.341467+0000", 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6860000000000001e-06 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.342907+0000", 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014395359999999999 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.342950+0000", 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3386000000000001e-05 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.21791605, 2024-09-16T02:14:34.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003304399, 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346336+0000", 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8929999999999999e-06 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346359+0000", 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2534000000000001e-05 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346435+0000", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6236000000000007e-05 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346451+0000", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5707000000000001e-05 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346735+0000", 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028424199999999998 2024-09-16T02:14:34.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346747+0000", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1725999999999999e-05 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349591+0000", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002844329 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349639+0000", 2024-09-16T02:14:34.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7732000000000003e-05 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.216406881, 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039104070000000003, 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.612 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347844+0000", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9999999999999999e-07 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347874+0000", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0167000000000001e-05 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347952+0000", 2024-09-16T02:14:34.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8427e-05 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347965+0000", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2937e-05 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348056+0000", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0390999999999996e-05 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348066+0000", 2024-09-16T02:14:34.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0584000000000001e-05 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351696+0000", 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003630084 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351754+0000", 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7516999999999998e-05 2024-09-16T02:14:34.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.21462857199999999, 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046999930000000004, 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8400000000000001e-07 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349656+0000", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4477999999999998e-05 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349726+0000", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9680999999999999e-05 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349739+0000", 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.323e-05 2024-09-16T02:14:34.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349814+0000", 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349826+0000", 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1955e-05 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354271+0000", 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044454020000000002 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354322+0000", 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0506000000000001e-05 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:34.621 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.207018968, 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052360729999999999, 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:34.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357232+0000", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357254+0000", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2855999999999999e-05 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:14:34.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5995000000000002e-05 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357322+0000", 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1798e-05 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357409+0000", 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7282999999999998e-05 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357425+0000", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.52e-05 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362424+0000", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049990690000000001 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362467+0000", 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3674000000000002e-05 2024-09-16T02:14:34.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.206886825, 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065655959999999999, 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357364+0000", 2024-09-16T02:14:34.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8199999999999999e-07 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357381+0000", 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7022999999999999e-05 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357419+0000", 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8779999999999998e-05 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357427+0000", 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357549+0000", 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121402 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357561+0000", 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2807e-05 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363880+0000", 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063186550000000003 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363929+0000", 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9030999999999999e-05 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:34.630 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.20463967499999999, 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045504229999999996, 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359612+0000", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.122e-06 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359626+0000", 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4422e-05 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359678+0000", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1765999999999999e-05 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359690+0000", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2143e-05 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359766+0000", 2024-09-16T02:14:34.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5883000000000005e-05 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359776+0000", 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0448999999999999e-05 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364111+0000", 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043348809999999996 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364161+0000", 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9756999999999998e-05 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.20434477700000001, 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065736739999999998, 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:34.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359931+0000", 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4975999999999999e-05 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359972+0000", 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0244000000000003e-05 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359982+0000", 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0710000000000001e-05 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360066+0000", 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3308999999999998e-05 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360080+0000", 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4363e-05 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366432+0000", 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00635214 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366479+0000", 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6959999999999998e-05 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/32 ECSubWrite(tid=9, reqid=osd.14.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=45'8))", 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367938+0000", 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.19631275100000001, 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001579268, 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367938+0000", 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367938+0000", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367939+0000", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367952+0000", 2024-09-16T02:14:34.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3036e-05 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368001+0000", 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8872000000000002e-05 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368011+0000", 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0524e-05 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368091+0000", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9332000000000005e-05 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368102+0000", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1837e-05 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369468+0000", 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013655570000000001 2024-09-16T02:14:34.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369517+0000", 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8877e-05 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=25, reqid=osd.2.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.19013627399999999, 2024-09-16T02:14:34.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026568659999999999, 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374115+0000", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3850000000000001e-06 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374134+0000", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8363999999999999e-05 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374183+0000", 2024-09-16T02:14:34.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9078e-05 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374195+0000", 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1704e-05 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374270+0000", 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999997e-05 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374283+0000", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2044e-05 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.376722+0000", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024396600000000002 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.376771+0000", 2024-09-16T02:14:34.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8721000000000003e-05 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=33, reqid=client.4598.0:257, at_version=45'23, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.389984+0000", 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.174266318, 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011527309999999999, 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.649 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.389984+0000", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.389984+0000", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.389986+0000", 2024-09-16T02:14:34.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5910000000000001e-06 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.390012+0000", 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6707e-05 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.390071+0000", 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8542999999999998e-05 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.390084+0000", 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3349e-05 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.390158+0000", 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4244000000000003e-05 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.390165+0000", 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8909999999999998e-06 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.391098+0000", 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093245399999999999 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.391137+0000", 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8952000000000002e-05 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=34, reqid=client.4598.0:257, at_version=0'0, trim_to=0'0, roll_forward_to=45'23))", 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.393630+0000", 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.17062063599999999, 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091608200000000003, 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393630+0000", 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393630+0000", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393631+0000", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1570000000000001e-06 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393645+0000", 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4295e-05 2024-09-16T02:14:34.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393694+0000", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8560999999999998e-05 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393705+0000", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1612e-05 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393777+0000", 2024-09-16T02:14:34.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1494000000000004e-05 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393789+0000", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.181e-05 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.394513+0000", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00072471099999999995 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.394546+0000", 2024-09-16T02:14:34.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2441999999999999e-05 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:34.658 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:34.660 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:34.656+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:34.675 INFO:tasks.rados.rados.0.smithi195.stdout:98: finishing write tid 3 to smithi19530025-8 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:98: finishing write tid 4 to smithi19530025-8 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:98: oid 8 updating version 0 to 17 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:98: oid 8 version 17 is already newer than 16 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:98: oid 8 version 17 is already newer than 15 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 8 v 17 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-16T02:14:34.677 INFO:tasks.rados.rados.0.smithi195.stdout:98: left oid 8 (ObjNum 61 snap 5 seq_num 61) 2024-09-16T02:14:34.702 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:34.703 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:34.722 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-16T02:14:34.340040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25035315,"num_objects":65,"num_object_clones":12,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":3,"num_read":209,"num_read_kb":2158,"num_write":272,"num_write_kb":27170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":490348,"kb_used_data":49508,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509459092,"statfs":{"total":1546188226560,"available":1545686110208,"internally_reserved":0,"allocated":50696192,"data_stored":55337213,"data_compressed":905710,"data_compressed_allocated":1273856,"data_compressed_original":3407872,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028308"},"pg_stats":[{"pgid":"3.f","version":"35'6","reported_seq":48,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.696089+0000","last_change":"2024-09-16T02:14:22.488124+0000","last_active":"2024-09-16T02:14:29.696089+0000","last_peered":"2024-09-16T02:14:29.696089+0000","last_clean":"2024-09-16T02:14:29.696089+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:14:29.696089+0000","last_undegraded":"2024-09-16T02:14:29.696089+0000","last_fullsized":"2024-09-16T02:14:29.696089+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.309266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015808,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":982,"num_write":6,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"36'11","reported_seq":78,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353752+0000","last_change":"2024-09-16T02:14:33.354131+0000","last_active":"2024-09-16T02:14:33.353752+0000","last_peered":"2024-09-16T02:14:33.353752+0000","last_clean":"2024-09-16T02:14:33.353752+0000","last_became_active":"2024-09-16T02:14:22.491985+0000","last_became_peered":"2024-09-16T02:14:22.491985+0000","last_unstale":"2024-09-16T02:14:33.353752+0000","last_undegraded":"2024-09-16T02:14:33.353752+0000","last_fullsized":"2024-09-16T02:14:33.353752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.130206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078863499999999999,"stat_sum":{"num_bytes":2015232,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":9,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,10,11,0,15],"acting":[8,9,1,10,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.d","version":"38'10","reported_seq":49,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303791+0000","last_change":"2024-09-16T02:14:22.477523+0000","last_active":"2024-09-16T02:14:29.303791+0000","last_peered":"2024-09-16T02:14:29.303791+0000","last_clean":"2024-09-16T02:14:29.303791+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:14:29.303791+0000","last_undegraded":"2024-09-16T02:14:29.303791+0000","last_fullsized":"2024-09-16T02:14:29.303791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:23.922436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1523712,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"36'6","reported_seq":70,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.352444+0000","last_change":"2024-09-16T02:14:33.353749+0000","last_active":"2024-09-16T02:14:33.352444+0000","last_peered":"2024-09-16T02:14:33.352444+0000","last_clean":"2024-09-16T02:14:33.352444+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:14:33.352444+0000","last_undegraded":"2024-09-16T02:14:33.352444+0000","last_fullsized":"2024-09-16T02:14:33.352444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.869333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019779519999999998,"stat_sum":{"num_bytes":278528,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.b","version":"35'6","reported_seq":43,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.961139+0000","last_change":"2024-09-16T02:14:22.494661+0000","last_active":"2024-09-16T02:14:29.961139+0000","last_peered":"2024-09-16T02:14:29.961139+0000","last_clean":"2024-09-16T02:14:29.961139+0000","last_became_active":"2024-09-16T02:14:22.493029+0000","last_became_peered":"2024-09-16T02:14:22.493029+0000","last_unstale":"2024-09-16T02:14:29.961139+0000","last_undegraded":"2024-09-16T02:14:29.961139+0000","last_fullsized":"2024-09-16T02:14:29.961139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:29.713683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,10,13],"acting":[7,15,1,9,8,10,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303729+0000","last_change":"2024-09-16T02:14:22.349289+0000","last_active":"2024-09-16T02:14:29.303729+0000","last_peered":"2024-09-16T02:14:29.303729+0000","last_clean":"2024-09-16T02:14:29.303729+0000","last_became_active":"2024-09-16T02:14:22.348067+0000","last_became_peered":"2024-09-16T02:14:22.348067+0000","last_unstale":"2024-09-16T02:14:29.303729+0000","last_undegraded":"2024-09-16T02:14:29.303729+0000","last_fullsized":"2024-09-16T02:14:29.303729+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.571559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1,14,8,15,2],"acting":[6,10,1,14,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"36'4","reported_seq":37,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.719545+0000","last_change":"2024-09-16T02:14:22.482602+0000","last_active":"2024-09-16T02:14:28.719545+0000","last_peered":"2024-09-16T02:14:28.719545+0000","last_clean":"2024-09-16T02:14:28.719545+0000","last_became_active":"2024-09-16T02:14:22.482031+0000","last_became_peered":"2024-09-16T02:14:22.482031+0000","last_unstale":"2024-09-16T02:14:28.719545+0000","last_undegraded":"2024-09-16T02:14:28.719545+0000","last_fullsized":"2024-09-16T02:14:28.719545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:35.795710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,15,4,14,11,2],"acting":[12,1,15,4,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"38'12","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960528+0000","last_change":"2024-09-16T02:14:22.487481+0000","last_active":"2024-09-16T02:14:29.960528+0000","last_peered":"2024-09-16T02:14:29.960528+0000","last_clean":"2024-09-16T02:14:29.960528+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:14:29.960528+0000","last_undegraded":"2024-09-16T02:14:29.960528+0000","last_fullsized":"2024-09-16T02:14:29.960528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.188205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637824,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":11,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":176,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302637+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:29.302637+0000","last_peered":"2024-09-16T02:14:29.302637+0000","last_clean":"2024-09-16T02:14:29.302637+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:29.302637+0000","last_undegraded":"2024-09-16T02:14:29.302637+0000","last_fullsized":"2024-09-16T02:14:29.302637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960491+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:29.960491+0000","last_peered":"2024-09-16T02:14:29.960491+0000","last_clean":"2024-09-16T02:14:29.960491+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:29.960491+0000","last_undegraded":"2024-09-16T02:14:29.960491+0000","last_fullsized":"2024-09-16T02:14:29.960491+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"36'20","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302919+0000","last_change":"2024-09-16T02:14:22.479623+0000","last_active":"2024-09-16T02:14:29.302919+0000","last_peered":"2024-09-16T02:14:29.302919+0000","last_clean":"2024-09-16T02:14:29.302919+0000","last_became_active":"2024-09-16T02:14:22.476669+0000","last_became_peered":"2024-09-16T02:14:22.476669+0000","last_unstale":"2024-09-16T02:14:29.302919+0000","last_undegraded":"2024-09-16T02:14:29.302919+0000","last_fullsized":"2024-09-16T02:14:29.302919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:42.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":20,"num_read_kb":933,"num_write":18,"num_write_kb":3936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,10,14,1,0],"acting":[13,12,6,10,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.695504+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:29.695504+0000","last_peered":"2024-09-16T02:14:29.695504+0000","last_clean":"2024-09-16T02:14:29.695504+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:29.695504+0000","last_undegraded":"2024-09-16T02:14:29.695504+0000","last_fullsized":"2024-09-16T02:14:29.695504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"38'21","reported_seq":60,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.301172+0000","last_change":"2024-09-16T02:14:22.494823+0000","last_active":"2024-09-16T02:14:28.301172+0000","last_peered":"2024-09-16T02:14:28.301172+0000","last_clean":"2024-09-16T02:14:28.301172+0000","last_became_active":"2024-09-16T02:14:22.493319+0000","last_became_peered":"2024-09-16T02:14:22.493319+0000","last_unstale":"2024-09-16T02:14:28.301172+0000","last_undegraded":"2024-09-16T02:14:28.301172+0000","last_fullsized":"2024-09-16T02:14:28.301172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:31.165101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3883008,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":18,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302695+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:29.302695+0000","last_peered":"2024-09-16T02:14:29.302695+0000","last_clean":"2024-09-16T02:14:29.302695+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:29.302695+0000","last_undegraded":"2024-09-16T02:14:29.302695+0000","last_fullsized":"2024-09-16T02:14:29.302695+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"35'2","reported_seq":36,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.719402+0000","last_change":"2024-09-16T02:14:22.474705+0000","last_active":"2024-09-16T02:14:28.719402+0000","last_peered":"2024-09-16T02:14:28.719402+0000","last_clean":"2024-09-16T02:14:28.719402+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:14:28.719402+0000","last_undegraded":"2024-09-16T02:14:28.719402+0000","last_fullsized":"2024-09-16T02:14:28.719402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:29.299216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.695571+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:29.695571+0000","last_peered":"2024-09-16T02:14:29.695571+0000","last_clean":"2024-09-16T02:14:29.695571+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:29.695571+0000","last_undegraded":"2024-09-16T02:14:29.695571+0000","last_fullsized":"2024-09-16T02:14:29.695571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"36'6","reported_seq":49,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302996+0000","last_change":"2024-09-16T02:14:22.486393+0000","last_active":"2024-09-16T02:14:29.302996+0000","last_peered":"2024-09-16T02:14:29.302996+0000","last_clean":"2024-09-16T02:14:29.302996+0000","last_became_active":"2024-09-16T02:14:22.484621+0000","last_became_peered":"2024-09-16T02:14:22.484621+0000","last_unstale":"2024-09-16T02:14:29.302996+0000","last_undegraded":"2024-09-16T02:14:29.302996+0000","last_fullsized":"2024-09-16T02:14:29.302996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:32.401332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475136,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":131,"num_write":5,"num_write_kb":464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,10],"acting":[13,2,11,14,4,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"36'11","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960046+0000","last_change":"2024-09-16T02:14:22.477726+0000","last_active":"2024-09-16T02:14:29.960046+0000","last_peered":"2024-09-16T02:14:29.960046+0000","last_clean":"2024-09-16T02:14:29.960046+0000","last_became_active":"2024-09-16T02:14:22.476388+0000","last_became_peered":"2024-09-16T02:14:22.476388+0000","last_unstale":"2024-09-16T02:14:29.960046+0000","last_undegraded":"2024-09-16T02:14:29.960046+0000","last_fullsized":"2024-09-16T02:14:29.960046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.807429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":4,"num_write":10,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,15,14,0,7,8],"acting":[10,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.301284+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:28.301284+0000","last_peered":"2024-09-16T02:14:28.301284+0000","last_clean":"2024-09-16T02:14:28.301284+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:28.301284+0000","last_undegraded":"2024-09-16T02:14:28.301284+0000","last_fullsized":"2024-09-16T02:14:28.301284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"36'8","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.961151+0000","last_change":"2024-09-16T02:14:22.348911+0000","last_active":"2024-09-16T02:14:29.961151+0000","last_peered":"2024-09-16T02:14:29.961151+0000","last_clean":"2024-09-16T02:14:29.961151+0000","last_became_active":"2024-09-16T02:14:22.346842+0000","last_became_peered":"2024-09-16T02:14:22.346842+0000","last_unstale":"2024-09-16T02:14:29.961151+0000","last_undegraded":"2024-09-16T02:14:29.961151+0000","last_fullsized":"2024-09-16T02:14:29.961151+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:28.345321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353186+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:33.353186+0000","last_peered":"2024-09-16T02:14:33.353186+0000","last_clean":"2024-09-16T02:14:33.353186+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:33.353186+0000","last_undegraded":"2024-09-16T02:14:33.353186+0000","last_fullsized":"2024-09-16T02:14:33.353186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'14","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960322+0000","last_change":"2024-09-16T02:14:22.496035+0000","last_active":"2024-09-16T02:14:29.960322+0000","last_peered":"2024-09-16T02:14:29.960322+0000","last_clean":"2024-09-16T02:14:29.960322+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:14:29.960322+0000","last_undegraded":"2024-09-16T02:14:29.960322+0000","last_fullsized":"2024-09-16T02:14:29.960322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:40.202676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":14,"num_write_kb":3632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.336952+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:33.336952+0000","last_peered":"2024-09-16T02:14:33.336952+0000","last_clean":"2024-09-16T02:14:33.336952+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:33.336952+0000","last_undegraded":"2024-09-16T02:14:33.336952+0000","last_fullsized":"2024-09-16T02:14:33.336952+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'4","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.337164+0000","last_change":"2024-09-16T02:14:33.337534+0000","last_active":"2024-09-16T02:14:33.337164+0000","last_peered":"2024-09-16T02:14:33.337164+0000","last_clean":"2024-09-16T02:14:33.337164+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:14:33.337164+0000","last_undegraded":"2024-09-16T02:14:33.337164+0000","last_fullsized":"2024-09-16T02:14:33.337164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.302344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080905199999999999,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303981+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:29.303981+0000","last_peered":"2024-09-16T02:14:29.303981+0000","last_clean":"2024-09-16T02:14:29.303981+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:29.303981+0000","last_undegraded":"2024-09-16T02:14:29.303981+0000","last_fullsized":"2024-09-16T02:14:29.303981+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":24444928,"num_objects":61,"num_object_clones":12,"num_object_copies":427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":3,"num_read":123,"num_read_kb":2085,"num_write":125,"num_write_kb":24496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43778048,"data_stored":50372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2600960,"data_stored":3011627,"data_compressed":84497,"data_compressed_allocated":114688,"data_compressed_original":659456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29724,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342116,"statfs":{"total":96636764160,"available":96606326784,"internally_reserved":0,"allocated":2760704,"data_stored":2968094,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33704,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338136,"statfs":{"total":96636764160,"available":96602251264,"internally_reserved":0,"allocated":2576384,"data_stored":2603550,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":5029888,"data_stored":5817119,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30532,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341308,"statfs":{"total":96636764160,"available":96605499392,"internally_reserved":0,"allocated":3579904,"data_stored":3873310,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313224,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31176,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340664,"statfs":{"total":96636764160,"available":96604839936,"internally_reserved":0,"allocated":4239360,"data_stored":4866847,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280520,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29472,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342368,"statfs":{"total":96636764160,"available":96606584832,"internally_reserved":0,"allocated":2498560,"data_stored":2697777,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29968,"kb_used_data":2940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341872,"statfs":{"total":96636764160,"available":96606076928,"internally_reserved":0,"allocated":3010560,"data_stored":3066398,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30524,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341316,"statfs":{"total":96636764160,"available":96605507584,"internally_reserved":0,"allocated":3575808,"data_stored":3842590,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29464,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342376,"statfs":{"total":96636764160,"available":96606593024,"internally_reserved":0,"allocated":2490368,"data_stored":2667295,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280520,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":4316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":4419584,"data_stored":5418046,"data_compressed":81565,"data_compressed_allocated":110592,"data_compressed_original":651264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30192,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341648,"statfs":{"total":96636764160,"available":96605847552,"internally_reserved":0,"allocated":3231744,"data_stored":3361310,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28696,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343144,"statfs":{"total":96636764160,"available":96607379456,"internally_reserved":0,"allocated":1708032,"data_stored":1788446,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30224,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341616,"statfs":{"total":96636764160,"available":96605814784,"internally_reserved":0,"allocated":3268608,"data_stored":3518987,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30004,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341836,"statfs":{"total":96636764160,"available":96606040064,"internally_reserved":0,"allocated":3043328,"data_stored":3068427,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29628,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342212,"statfs":{"total":96636764160,"available":96606425088,"internally_reserved":0,"allocated":2662400,"data_stored":2767390,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3948544,"data_stored":4657152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3170304,"data_stored":3641344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2084864,"data_stored":2465792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2822144,"data_stored":3129344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1298432,"data_stored":1556480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2600960,"data_stored":2834432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2846720,"data_stored":3276800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3149824,"data_stored":3600384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2019328,"data_stored":2383872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2621440,"data_stored":2826240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2252800,"data_stored":2535424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2166784,"data_stored":2371584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2351104,"data_stored":2736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":2117632,"data_stored":2240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4558848,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3768320,"data_stored":4583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:34.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:34.760 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:34.758+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:34.787 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:34.788 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:34.788 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:34.788 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:34.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:34.861 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:34.856+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:34.961 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:34.954+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:34.968 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:34.969 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:34.969 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:34.969 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:34.969 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:34.969 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:34.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:34.999 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:34.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:35.040 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:35.040 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:35.063 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:35.056+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:35.066 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:35.066 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:35.104 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:35.121 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:35.121 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:35.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:35.121 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.841197644999999, 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002207437, 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:35.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278078+0000", 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.15e-07 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278223+0000", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014522 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278313+0000", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9775e-05 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278329+0000", 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6232999999999999e-05 2024-09-16T02:14:35.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278450+0000", 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120557 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278461+0000", 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1697e-05 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.280237+0000", 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001775151 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.280285+0000", 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8389e-05 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/32 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:35.126 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.346493+0000", 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.772781921, 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017402629999999999, 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346493+0000", 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346493+0000", 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346493+0000", 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4299999999999999e-07 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346522+0000", 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8263999999999999e-05 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346580+0000", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8328999999999997e-05 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346592+0000", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1593000000000001e-05 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346684+0000", 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2058999999999994e-05 2024-09-16T02:14:35.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.346695+0000", 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.134e-05 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.348133+0000", 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001438426 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.348233+0000", 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100009 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/32 ECSubWrite(tid=31, reqid=osd.3.0:31, at_version=44'28, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.7718762139999999, 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049741270000000001, 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.131 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347400+0000", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3600000000000002e-07 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347420+0000", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0194e-05 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347461+0000", 2024-09-16T02:14:35.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0744999999999997e-05 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347469+0000", 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.96e-06 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347553+0000", 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4555000000000006e-05 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347565+0000", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1562999999999999e-05 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352277+0000", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047123479999999999 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352373+0000", 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5926000000000003e-05 2024-09-16T02:14:35.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/32 ECSubWrite(tid=32, reqid=osd.3.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.7544236849999999, 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032542029999999998, 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364852+0000", 2024-09-16T02:14:35.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3799999999999999e-07 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364857+0000", 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7769999999999999e-06 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364922+0000", 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5023000000000004e-05 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364932+0000", 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0078000000000001e-05 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.365047+0000", 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115016 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.365065+0000", 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7011e-05 2024-09-16T02:14:35.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.368063+0000", 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029989909999999999 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.368106+0000", 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2169000000000002e-05 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.76288995299999995, 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049611000000000004, 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:35.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9999999999999999e-07 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356486+0000", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100728 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356545+0000", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8567999999999999e-05 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356559+0000", 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3883000000000001e-05 2024-09-16T02:14:35.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356630+0000", 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1431000000000001e-05 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356642+0000", 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2418999999999999e-05 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.361297+0000", 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046547230000000004 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.361346+0000", 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9147999999999998e-05 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.146 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=34, reqid=osd.3.0:34, at_version=45'32, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75669468399999995, 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070140769999999996, 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362582+0000", 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-06 2024-09-16T02:14:35.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362610+0000", 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7945000000000001e-05 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362658+0000", 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8439000000000002e-05 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362670+0000", 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1960000000000001e-05 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362748+0000", 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8162000000000001e-05 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362759+0000", 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0978999999999999e-05 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369332+0000", 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065723960000000003 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369595+0000", 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026293399999999999 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.151 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(osd.11.22:2 3.es0 3:701f1021:::smithi19530025-33:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75429381200000001, 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042977079999999999, 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364984+0000", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1950000000000002e-06 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364993+0000", 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2110000000000004e-06 2024-09-16T02:14:35.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365019+0000", 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6511000000000001e-05 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365031+0000", 2024-09-16T02:14:35.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1333e-05 2024-09-16T02:14:35.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365119+0000", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8095000000000003e-05 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365234+0000", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115127 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369279+0000", 2024-09-16T02:14:35.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040452359999999998 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.es0 45/32 ECSubRead(tid=18, to_read={3:701f1021:::smithi19530025-33:head=0,135168,0}, subchunks={3:701f1021:::smithi19530025-33:head=[0,1]}, attrs_to_read=))", 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.365510+0000", 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75376516699999996, 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085826900000000005, 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.157 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365510+0000", 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365510+0000", 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365510+0000", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365510+0000", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365537+0000", 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6704999999999999e-05 2024-09-16T02:14:35.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365548+0000", 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1440999999999999e-05 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365645+0000", 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7268000000000005e-05 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366368+0000", 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000722855 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpReadReply(3.es0 45/32 ECSubReadReply(tid=18, attrs_read=0))", 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367378+0000", 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75189690899999995, 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058063199999999998, 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.161 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367378+0000", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367378+0000", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367379+0000", 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0470000000000001e-06 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367721+0000", 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000341744 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367791+0000", 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9849000000000002e-05 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367796+0000", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5470000000000004e-06 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367891+0000", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4704999999999995e-05 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367959+0000", 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7739999999999999e-05 2024-09-16T02:14:35.164 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=17, last_complete=45'13, committed=1, applied=1))", 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367484+0000", 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75179130800000005, 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036938900000000002, 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367484+0000", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367484+0000", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367484+0000", 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.01e-07 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367490+0000", 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8869999999999997e-06 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367744+0000", 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025368899999999998 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367753+0000", 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2609999999999999e-06 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367827+0000", 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4385e-05 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367853+0000", 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5766e-05 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.168 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75177991200000005, 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050183099999999998, 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1799999999999999e-07 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367880+0000", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038417900000000001 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368076+0000", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019593799999999999 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368085+0000", 2024-09-16T02:14:35.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4749999999999999e-06 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368154+0000", 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9239000000000006e-05 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368166+0000", 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2055999999999999e-05 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.372477+0000", 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043104889999999998 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.372514+0000", 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6715999999999999e-05 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.173 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpReadReply(3.es0 45/32 ECSubReadReply(tid=18, attrs_read=0))", 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367882+0000", 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75139265700000002, 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045888599999999998, 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367882+0000", 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367882+0000", 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367883+0000", 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8699999999999995e-07 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368014+0000", 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013105699999999999 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368112+0000", 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7879000000000003e-05 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368117+0000", 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7999999999999998e-06 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368286+0000", 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000168832 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368341+0000", 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5730999999999999e-05 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.177 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpReadReply(3.es0 45/32 ECSubReadReply(tid=18, attrs_read=0))", 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.368455+0000", 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.75082060900000003, 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084132300000000005, 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368455+0000", 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368455+0000", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368455+0000", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6499999999999998e-07 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368809+0000", 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035317500000000001 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368850+0000", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1180000000000002e-05 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368858+0000", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8699999999999992e-06 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368920+0000", 2024-09-16T02:14:35.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2151999999999997e-05 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369296+0000", 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037608100000000001 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.es0 45/32 ECSubWriteReply(tid=17, last_complete=45'13, committed=1, applied=1))", 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.370488+0000", 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.74878681400000002, 2024-09-16T02:14:35.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00079784300000000001, 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370488+0000", 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370488+0000", 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370489+0000", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5899999999999995e-07 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370494+0000", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0860000000000001e-06 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370530+0000", 2024-09-16T02:14:35.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5667000000000003e-05 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370539+0000", 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5730000000000008e-06 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370605+0000", 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5989999999999997e-05 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.371286+0000", 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068166799999999997 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=45'32))", 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.74192275799999996, 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027598050000000002, 2024-09-16T02:14:35.185 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377353+0000", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8200000000000001e-07 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377365+0000", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2094e-05 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377467+0000", 2024-09-16T02:14:35.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102618 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377478+0000", 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0879999999999999e-05 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377694+0000", 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021557799999999999 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377707+0000", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3237999999999999e-05 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380056+0000", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002349425 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380112+0000", 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5689999999999997e-05 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.189 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=22, reqid=osd.10.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.73872376399999995, 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022536359999999998, 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380552+0000", 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.47e-07 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380562+0000", 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.448e-06 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380602+0000", 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9894999999999997e-05 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380612+0000", 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8649999999999996e-06 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380666+0000", 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4789e-05 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380674+0000", 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.382761+0000", 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020867749999999999 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.382805+0000", 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4218000000000003e-05 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.194 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=23, reqid=client.4598.0:253, at_version=45'14, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.387723+0000", 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.73155211499999995, 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013090879999999999, 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.387723+0000", 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.387723+0000", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.387724+0000", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3060000000000001e-06 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388005+0000", 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028029800000000001 2024-09-16T02:14:35.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388060+0000", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4911000000000003e-05 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388073+0000", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3927999999999999e-05 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388149+0000", 2024-09-16T02:14:35.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5053000000000001e-05 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388156+0000", 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7200000000000006e-06 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.388989+0000", 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00083284500000000003 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.389032+0000", 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3027e-05 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=24, reqid=client.4598.0:254, at_version=45'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.392345+0000", 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.72692972199999994, 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00103794, 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392345+0000", 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392345+0000", 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392346+0000", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.33e-07 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392503+0000", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015677599999999999 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392541+0000", 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8358000000000001e-05 2024-09-16T02:14:35.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392551+0000", 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8139999999999995e-06 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392619+0000", 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7527999999999994e-05 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.392628+0000", 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6139999999999998e-06 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393335+0000", 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070661900000000002 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.393383+0000", 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8297999999999997e-05 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.396698+0000", 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.72257729500000001, 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074432600000000002, 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396698+0000", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396698+0000", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396699+0000", 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4300000000000002e-07 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396714+0000", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4918e-05 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396747+0000", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3124000000000003e-05 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396753+0000", 2024-09-16T02:14:35.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0390000000000001e-06 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396815+0000", 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2045999999999995e-05 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.396826+0000", 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1032e-05 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.397399+0000", 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057323200000000002 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.397442+0000", 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3092e-05 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=26, reqid=client.4598.0:255, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:35.208 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.675681+0000", 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.44359421199999999, 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091076799999999995, 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675681+0000", 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675681+0000", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675682+0000", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5120000000000001e-06 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675698+0000", 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5258e-05 2024-09-16T02:14:35.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675750+0000", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2609999999999999e-05 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675764+0000", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3691e-05 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675852+0000", 2024-09-16T02:14:35.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8389999999999999e-05 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.675865+0000", 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2608e-05 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.676524+0000", 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00065886700000000005 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:35.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.676592+0000", 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7831999999999997e-05 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:35.213 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:35.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:35.215 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:35.215 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:35.216 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:35.217 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:35.217 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-16T02:14:34.340040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25035315,"num_objects":65,"num_object_clones":12,"num_object_copies":435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":3,"num_read":209,"num_read_kb":2158,"num_write":272,"num_write_kb":27170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":256,"ondisk_log_size":256,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":490348,"kb_used_data":49508,"kb_used_omap":24,"kb_used_meta":440423,"kb_avail":1509459092,"statfs":{"total":1546188226560,"available":1545686110208,"internally_reserved":0,"allocated":50696192,"data_stored":55337213,"data_compressed":905710,"data_compressed_allocated":1273856,"data_compressed_original":3407872,"omap_allocated":25522,"internal_metadata":450993230},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028308"},"pg_stats":[{"pgid":"3.f","version":"35'6","reported_seq":48,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.696089+0000","last_change":"2024-09-16T02:14:22.488124+0000","last_active":"2024-09-16T02:14:29.696089+0000","last_peered":"2024-09-16T02:14:29.696089+0000","last_clean":"2024-09-16T02:14:29.696089+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:14:29.696089+0000","last_undegraded":"2024-09-16T02:14:29.696089+0000","last_fullsized":"2024-09-16T02:14:29.696089+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.309266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1015808,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":982,"num_write":6,"num_write_kb":992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"36'11","reported_seq":78,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353752+0000","last_change":"2024-09-16T02:14:33.354131+0000","last_active":"2024-09-16T02:14:33.353752+0000","last_peered":"2024-09-16T02:14:33.353752+0000","last_clean":"2024-09-16T02:14:33.353752+0000","last_became_active":"2024-09-16T02:14:22.491985+0000","last_became_peered":"2024-09-16T02:14:22.491985+0000","last_unstale":"2024-09-16T02:14:33.353752+0000","last_undegraded":"2024-09-16T02:14:33.353752+0000","last_fullsized":"2024-09-16T02:14:33.353752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.130206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078863499999999999,"stat_sum":{"num_bytes":2015232,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":9,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,10,11,0,15],"acting":[8,9,1,10,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.d","version":"38'10","reported_seq":49,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303791+0000","last_change":"2024-09-16T02:14:22.477523+0000","last_active":"2024-09-16T02:14:29.303791+0000","last_peered":"2024-09-16T02:14:29.303791+0000","last_clean":"2024-09-16T02:14:29.303791+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:14:29.303791+0000","last_undegraded":"2024-09-16T02:14:29.303791+0000","last_fullsized":"2024-09-16T02:14:29.303791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:23.922436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1523712,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"36'6","reported_seq":70,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.352444+0000","last_change":"2024-09-16T02:14:33.353749+0000","last_active":"2024-09-16T02:14:33.352444+0000","last_peered":"2024-09-16T02:14:33.352444+0000","last_clean":"2024-09-16T02:14:33.352444+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:14:33.352444+0000","last_undegraded":"2024-09-16T02:14:33.352444+0000","last_fullsized":"2024-09-16T02:14:33.352444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.869333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019779519999999998,"stat_sum":{"num_bytes":278528,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.b","version":"35'6","reported_seq":43,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.961139+0000","last_change":"2024-09-16T02:14:22.494661+0000","last_active":"2024-09-16T02:14:29.961139+0000","last_peered":"2024-09-16T02:14:29.961139+0000","last_clean":"2024-09-16T02:14:29.961139+0000","last_became_active":"2024-09-16T02:14:22.493029+0000","last_became_peered":"2024-09-16T02:14:22.493029+0000","last_unstale":"2024-09-16T02:14:29.961139+0000","last_undegraded":"2024-09-16T02:14:29.961139+0000","last_fullsized":"2024-09-16T02:14:29.961139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:29.713683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":6,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,10,13],"acting":[7,15,1,9,8,10,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":36,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303729+0000","last_change":"2024-09-16T02:14:22.349289+0000","last_active":"2024-09-16T02:14:29.303729+0000","last_peered":"2024-09-16T02:14:29.303729+0000","last_clean":"2024-09-16T02:14:29.303729+0000","last_became_active":"2024-09-16T02:14:22.348067+0000","last_became_peered":"2024-09-16T02:14:22.348067+0000","last_unstale":"2024-09-16T02:14:29.303729+0000","last_undegraded":"2024-09-16T02:14:29.303729+0000","last_fullsized":"2024-09-16T02:14:29.303729+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.571559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1,14,8,15,2],"acting":[6,10,1,14,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"36'4","reported_seq":37,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.719545+0000","last_change":"2024-09-16T02:14:22.482602+0000","last_active":"2024-09-16T02:14:28.719545+0000","last_peered":"2024-09-16T02:14:28.719545+0000","last_clean":"2024-09-16T02:14:28.719545+0000","last_became_active":"2024-09-16T02:14:22.482031+0000","last_became_peered":"2024-09-16T02:14:22.482031+0000","last_unstale":"2024-09-16T02:14:28.719545+0000","last_undegraded":"2024-09-16T02:14:28.719545+0000","last_fullsized":"2024-09-16T02:14:28.719545+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:35.795710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,15,4,14,11,2],"acting":[12,1,15,4,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"3.8","version":"38'12","reported_seq":53,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960528+0000","last_change":"2024-09-16T02:14:22.487481+0000","last_active":"2024-09-16T02:14:29.960528+0000","last_peered":"2024-09-16T02:14:29.960528+0000","last_clean":"2024-09-16T02:14:29.960528+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:14:29.960528+0000","last_undegraded":"2024-09-16T02:14:29.960528+0000","last_fullsized":"2024-09-16T02:14:29.960528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.188205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2637824,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":5,"num_write":11,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":176,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302637+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:29.302637+0000","last_peered":"2024-09-16T02:14:29.302637+0000","last_clean":"2024-09-16T02:14:29.302637+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:29.302637+0000","last_undegraded":"2024-09-16T02:14:29.302637+0000","last_fullsized":"2024-09-16T02:14:29.302637+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960491+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:29.960491+0000","last_peered":"2024-09-16T02:14:29.960491+0000","last_clean":"2024-09-16T02:14:29.960491+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:29.960491+0000","last_undegraded":"2024-09-16T02:14:29.960491+0000","last_fullsized":"2024-09-16T02:14:29.960491+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"36'20","reported_seq":66,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302919+0000","last_change":"2024-09-16T02:14:22.479623+0000","last_active":"2024-09-16T02:14:29.302919+0000","last_peered":"2024-09-16T02:14:29.302919+0000","last_clean":"2024-09-16T02:14:29.302919+0000","last_became_active":"2024-09-16T02:14:22.476669+0000","last_became_peered":"2024-09-16T02:14:22.476669+0000","last_unstale":"2024-09-16T02:14:29.302919+0000","last_undegraded":"2024-09-16T02:14:29.302919+0000","last_fullsized":"2024-09-16T02:14:29.302919+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:42.576568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4030464,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":20,"num_read_kb":933,"num_write":18,"num_write_kb":3936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,10,14,1,0],"acting":[13,12,6,10,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":43,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.695504+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:29.695504+0000","last_peered":"2024-09-16T02:14:29.695504+0000","last_clean":"2024-09-16T02:14:29.695504+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:29.695504+0000","last_undegraded":"2024-09-16T02:14:29.695504+0000","last_fullsized":"2024-09-16T02:14:29.695504+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"38'21","reported_seq":60,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.301172+0000","last_change":"2024-09-16T02:14:22.494823+0000","last_active":"2024-09-16T02:14:28.301172+0000","last_peered":"2024-09-16T02:14:28.301172+0000","last_clean":"2024-09-16T02:14:28.301172+0000","last_became_active":"2024-09-16T02:14:22.493319+0000","last_became_peered":"2024-09-16T02:14:22.493319+0000","last_unstale":"2024-09-16T02:14:28.301172+0000","last_undegraded":"2024-09-16T02:14:28.301172+0000","last_fullsized":"2024-09-16T02:14:28.301172+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:31.165101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3883008,"num_objects":9,"num_object_clones":2,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":18,"num_write_kb":3792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302695+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:29.302695+0000","last_peered":"2024-09-16T02:14:29.302695+0000","last_clean":"2024-09-16T02:14:29.302695+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:29.302695+0000","last_undegraded":"2024-09-16T02:14:29.302695+0000","last_fullsized":"2024-09-16T02:14:29.302695+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"35'2","reported_seq":36,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.719402+0000","last_change":"2024-09-16T02:14:22.474705+0000","last_active":"2024-09-16T02:14:28.719402+0000","last_peered":"2024-09-16T02:14:28.719402+0000","last_clean":"2024-09-16T02:14:28.719402+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:14:28.719402+0000","last_undegraded":"2024-09-16T02:14:28.719402+0000","last_fullsized":"2024-09-16T02:14:28.719402+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:29.299216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":2,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.695571+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:29.695571+0000","last_peered":"2024-09-16T02:14:29.695571+0000","last_clean":"2024-09-16T02:14:29.695571+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:29.695571+0000","last_undegraded":"2024-09-16T02:14:29.695571+0000","last_fullsized":"2024-09-16T02:14:29.695571+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"36'6","reported_seq":49,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.302996+0000","last_change":"2024-09-16T02:14:22.486393+0000","last_active":"2024-09-16T02:14:29.302996+0000","last_peered":"2024-09-16T02:14:29.302996+0000","last_clean":"2024-09-16T02:14:29.302996+0000","last_became_active":"2024-09-16T02:14:22.484621+0000","last_became_peered":"2024-09-16T02:14:22.484621+0000","last_unstale":"2024-09-16T02:14:29.302996+0000","last_undegraded":"2024-09-16T02:14:29.302996+0000","last_fullsized":"2024-09-16T02:14:29.302996+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:32.401332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475136,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":131,"num_write":5,"num_write_kb":464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,10],"acting":[13,2,11,14,4,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"36'11","reported_seq":51,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960046+0000","last_change":"2024-09-16T02:14:22.477726+0000","last_active":"2024-09-16T02:14:29.960046+0000","last_peered":"2024-09-16T02:14:29.960046+0000","last_clean":"2024-09-16T02:14:29.960046+0000","last_became_active":"2024-09-16T02:14:22.476388+0000","last_became_peered":"2024-09-16T02:14:22.476388+0000","last_unstale":"2024-09-16T02:14:29.960046+0000","last_undegraded":"2024-09-16T02:14:29.960046+0000","last_fullsized":"2024-09-16T02:14:29.960046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.807429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1441792,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":4,"num_write":10,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,15,14,0,7,8],"acting":[10,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":39,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-16T02:14:28.301284+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:28.301284+0000","last_peered":"2024-09-16T02:14:28.301284+0000","last_clean":"2024-09-16T02:14:28.301284+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:28.301284+0000","last_undegraded":"2024-09-16T02:14:28.301284+0000","last_fullsized":"2024-09-16T02:14:28.301284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"36'8","reported_seq":45,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.961151+0000","last_change":"2024-09-16T02:14:22.348911+0000","last_active":"2024-09-16T02:14:29.961151+0000","last_peered":"2024-09-16T02:14:29.961151+0000","last_clean":"2024-09-16T02:14:29.961151+0000","last_became_active":"2024-09-16T02:14:22.346842+0000","last_became_peered":"2024-09-16T02:14:22.346842+0000","last_unstale":"2024-09-16T02:14:29.961151+0000","last_undegraded":"2024-09-16T02:14:29.961151+0000","last_fullsized":"2024-09-16T02:14:29.961151+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:28.345321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353186+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:33.353186+0000","last_peered":"2024-09-16T02:14:33.353186+0000","last_clean":"2024-09-16T02:14:33.353186+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:33.353186+0000","last_undegraded":"2024-09-16T02:14:33.353186+0000","last_fullsized":"2024-09-16T02:14:33.353186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"35'14","reported_seq":57,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.960322+0000","last_change":"2024-09-16T02:14:22.496035+0000","last_active":"2024-09-16T02:14:29.960322+0000","last_peered":"2024-09-16T02:14:29.960322+0000","last_clean":"2024-09-16T02:14:29.960322+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:14:29.960322+0000","last_undegraded":"2024-09-16T02:14:29.960322+0000","last_fullsized":"2024-09-16T02:14:29.960322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:40.202676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3145728,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":14,"num_write_kb":3632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.336952+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:33.336952+0000","last_peered":"2024-09-16T02:14:33.336952+0000","last_clean":"2024-09-16T02:14:33.336952+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:33.336952+0000","last_undegraded":"2024-09-16T02:14:33.336952+0000","last_fullsized":"2024-09-16T02:14:33.336952+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'4","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.337164+0000","last_change":"2024-09-16T02:14:33.337534+0000","last_active":"2024-09-16T02:14:33.337164+0000","last_peered":"2024-09-16T02:14:33.337164+0000","last_clean":"2024-09-16T02:14:33.337164+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:14:33.337164+0000","last_undegraded":"2024-09-16T02:14:33.337164+0000","last_fullsized":"2024-09-16T02:14:33.337164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.302344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080905199999999999,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-16T02:14:29.303981+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:29.303981+0000","last_peered":"2024-09-16T02:14:29.303981+0000","last_clean":"2024-09-16T02:14:29.303981+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:29.303981+0000","last_undegraded":"2024-09-16T02:14:29.303981+0000","last_fullsized":"2024-09-16T02:14:29.303981+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":24444928,"num_objects":61,"num_object_clones":12,"num_object_copies":427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":3,"num_read":123,"num_read_kb":2085,"num_write":125,"num_write_kb":24496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43778048,"data_stored":50372608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33740,"kb_used_data":2540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338100,"statfs":{"total":96636764160,"available":96602214400,"internally_reserved":0,"allocated":2600960,"data_stored":3011627,"data_compressed":84497,"data_compressed_allocated":114688,"data_compressed_original":659456,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29724,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342116,"statfs":{"total":96636764160,"available":96606326784,"internally_reserved":0,"allocated":2760704,"data_stored":2968094,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33704,"kb_used_data":2516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338136,"statfs":{"total":96636764160,"available":96602251264,"internally_reserved":0,"allocated":2576384,"data_stored":2603550,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":5029888,"data_stored":5817119,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30532,"kb_used_data":3496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341308,"statfs":{"total":96636764160,"available":96605499392,"internally_reserved":0,"allocated":3579904,"data_stored":3873310,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313224,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31176,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340664,"statfs":{"total":96636764160,"available":96604839936,"internally_reserved":0,"allocated":4239360,"data_stored":4866847,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280520,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29472,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342368,"statfs":{"total":96636764160,"available":96606584832,"internally_reserved":0,"allocated":2498560,"data_stored":2697777,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280520,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29968,"kb_used_data":2940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341872,"statfs":{"total":96636764160,"available":96606076928,"internally_reserved":0,"allocated":3010560,"data_stored":3066398,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280519,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30524,"kb_used_data":3492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341316,"statfs":{"total":96636764160,"available":96605507584,"internally_reserved":0,"allocated":3575808,"data_stored":3842590,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29464,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342376,"statfs":{"total":96636764160,"available":96606593024,"internally_reserved":0,"allocated":2490368,"data_stored":2667295,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280520,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31352,"kb_used_data":4316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340488,"statfs":{"total":96636764160,"available":96604659712,"internally_reserved":0,"allocated":4419584,"data_stored":5418046,"data_compressed":81565,"data_compressed_allocated":110592,"data_compressed_original":651264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30192,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341648,"statfs":{"total":96636764160,"available":96605847552,"internally_reserved":0,"allocated":3231744,"data_stored":3361310,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28696,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343144,"statfs":{"total":96636764160,"available":96607379456,"internally_reserved":0,"allocated":1708032,"data_stored":1788446,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280520,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30224,"kb_used_data":3192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341616,"statfs":{"total":96636764160,"available":96605814784,"internally_reserved":0,"allocated":3268608,"data_stored":3518987,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280520,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30004,"kb_used_data":2972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341836,"statfs":{"total":96636764160,"available":96606040064,"internally_reserved":0,"allocated":3043328,"data_stored":3068427,"data_compressed":51966,"data_compressed_allocated":73728,"data_compressed_original":147456,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280520,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29628,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342212,"statfs":{"total":96636764160,"available":96606425088,"internally_reserved":0,"allocated":2662400,"data_stored":2767390,"data_compressed":49034,"data_compressed_allocated":69632,"data_compressed_original":139264,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3948544,"data_stored":4657152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3170304,"data_stored":3641344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2084864,"data_stored":2465792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2822144,"data_stored":3129344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1298432,"data_stored":1556480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2600960,"data_stored":2834432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2846720,"data_stored":3276800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3149824,"data_stored":3600384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2019328,"data_stored":2383872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2621440,"data_stored":2826240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2252800,"data_stored":2535424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2166784,"data_stored":2371584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2351104,"data_stored":2736128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":2117632,"data_stored":2240512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4558848,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3768320,"data_stored":4583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:35.219 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T02:14:35.219 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:14:35.220 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T02:14:35.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:35.220 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:35.214+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:35.270 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:35.270 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:35.296 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:35.297 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:35.315 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:35.310+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:35.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:35.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:35.353 INFO:tasks.rados.rados.0.smithi195.stdout:98: done (1 left) 2024-09-16T02:14:35.354 INFO:tasks.rados.rados.0.smithi195.stdout:100: done (0 left) 2024-09-16T02:14:35.354 INFO:tasks.rados.rados.0.smithi195.stdout:101: setattr oid 48 current snap is 6 2024-09-16T02:14:35.356 INFO:tasks.rados.rados.0.smithi195.stdout:102: copy_from oid 14 from oid 10 current snap is 6 2024-09-16T02:14:35.356 INFO:tasks.rados.rados.0.smithi195.stdout:103: rollback oid 18 current snap is 6 2024-09-16T02:14:35.356 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 18 to 5 2024-09-16T02:14:35.356 INFO:tasks.rados.rados.0.smithi195.stdout:104: snap_create 2024-09-16T02:14:35.361 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:35.361 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:35.373 INFO:tasks.rados.rados.0.smithi195.stdout:103: finishing rollback tid 0 to smithi19530025-18 2024-09-16T02:14:35.374 INFO:tasks.rados.rados.0.smithi195.stdout:103: finishing rollback tid 1 to smithi19530025-18 2024-09-16T02:14:35.381 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 51 (ObjNum 59 snap 5 seq_num 59) dirty exists 2024-09-16T02:14:35.383 INFO:tasks.rados.rados.0.smithi195.stdout:102: finishing copy_from racing read to smithi19530025-14 2024-09-16T02:14:35.402 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:35.402 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:35.402 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:35.402 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:35.416 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:35.411+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:35.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:14:35.517 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:35.512+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:35.518 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:35.518 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:35.551 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:35.552 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:35.582 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:35.582 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:35.584 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:35.584 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:35.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:35.586 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:35.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:14:35.607 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-16T02:14:35.349591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24724019,"num_objects":50,"num_object_clones":2,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":240,"num_read_kb":6902,"num_write":334,"num_write_kb":33170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":365,"ondisk_log_size":365,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":499500,"kb_used_data":54516,"kb_used_omap":24,"kb_used_meta":444583,"kb_avail":1509449940,"statfs":{"total":1546188226560,"available":1545676738560,"internally_reserved":0,"allocated":55824384,"data_stored":59875487,"data_compressed":1112963,"data_compressed_allocated":1560576,"data_compressed_original":3981312,"omap_allocated":25522,"internal_metadata":455253070},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-311296,"num_objects":-15,"num_object_clones":-10,"num_object_copies":-105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":-3,"num_read":31,"num_read_kb":4744,"num_write":62,"num_write_kb":6000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.031970"},"pg_stats":[{"pgid":"3.f","version":"45'16","reported_seq":93,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.374929+0000","last_change":"2024-09-16T02:14:34.376747+0000","last_active":"2024-09-16T02:14:34.374929+0000","last_peered":"2024-09-16T02:14:34.374929+0000","last_clean":"2024-09-16T02:14:34.374929+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:14:34.374929+0000","last_undegraded":"2024-09-16T02:14:34.374929+0000","last_fullsized":"2024-09-16T02:14:34.374929+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:43.758899+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.029787456,"stat_sum":{"num_bytes":1228800,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":982,"num_write":7,"num_write_kb":1616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.e","version":"36'11","reported_seq":78,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353752+0000","last_change":"2024-09-16T02:14:33.354131+0000","last_active":"2024-09-16T02:14:33.353752+0000","last_peered":"2024-09-16T02:14:33.353752+0000","last_clean":"2024-09-16T02:14:33.353752+0000","last_became_active":"2024-09-16T02:14:22.491985+0000","last_became_peered":"2024-09-16T02:14:22.491985+0000","last_unstale":"2024-09-16T02:14:33.353752+0000","last_undegraded":"2024-09-16T02:14:33.353752+0000","last_fullsized":"2024-09-16T02:14:33.353752+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.130206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078863499999999999,"stat_sum":{"num_bytes":2015232,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":9,"num_write_kb":1968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,10,11,0,15],"acting":[8,9,1,10,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.d","version":"45'16","reported_seq":90,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.351242+0000","last_change":"2024-09-16T02:14:34.354908+0000","last_active":"2024-09-16T02:14:34.351242+0000","last_peered":"2024-09-16T02:14:34.351242+0000","last_clean":"2024-09-16T02:14:34.351242+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:14:34.351242+0000","last_undegraded":"2024-09-16T02:14:34.351242+0000","last_fullsized":"2024-09-16T02:14:34.351242+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.437785+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012884852,"stat_sum":{"num_bytes":1523712,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":8,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.c","version":"36'6","reported_seq":70,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.352444+0000","last_change":"2024-09-16T02:14:33.353749+0000","last_active":"2024-09-16T02:14:33.352444+0000","last_peered":"2024-09-16T02:14:33.352444+0000","last_clean":"2024-09-16T02:14:33.352444+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:14:33.352444+0000","last_undegraded":"2024-09-16T02:14:33.352444+0000","last_fullsized":"2024-09-16T02:14:33.352444+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.869333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019779519999999998,"stat_sum":{"num_bytes":278528,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"3.b","version":"35'6","reported_seq":81,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.347362+0000","last_change":"2024-09-16T02:14:34.348590+0000","last_active":"2024-09-16T02:14:34.347362+0000","last_peered":"2024-09-16T02:14:34.347362+0000","last_clean":"2024-09-16T02:14:34.347362+0000","last_became_active":"2024-09-16T02:14:22.493029+0000","last_became_peered":"2024-09-16T02:14:22.493029+0000","last_unstale":"2024-09-16T02:14:34.347362+0000","last_undegraded":"2024-09-16T02:14:34.347362+0000","last_fullsized":"2024-09-16T02:14:34.347362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.284208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020161990000000002,"stat_sum":{"num_bytes":1048576,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":1297,"num_write":6,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,10,13],"acting":[7,15,1,9,8,10,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.a","version":"0'0","reported_seq":71,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.342325+0000","last_change":"2024-09-16T02:14:34.343148+0000","last_active":"2024-09-16T02:14:34.342325+0000","last_peered":"2024-09-16T02:14:34.342325+0000","last_clean":"2024-09-16T02:14:34.342325+0000","last_became_active":"2024-09-16T02:14:22.348067+0000","last_became_peered":"2024-09-16T02:14:22.348067+0000","last_unstale":"2024-09-16T02:14:34.342325+0000","last_undegraded":"2024-09-16T02:14:34.342325+0000","last_fullsized":"2024-09-16T02:14:34.342325+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:22.859825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001511709,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1,14,8,15,2],"acting":[6,10,1,14,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.9","version":"45'6","reported_seq":77,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.358692+0000","last_change":"2024-09-16T02:14:34.361442+0000","last_active":"2024-09-16T02:14:34.358692+0000","last_peered":"2024-09-16T02:14:34.358692+0000","last_clean":"2024-09-16T02:14:34.358692+0000","last_became_active":"2024-09-16T02:14:22.482031+0000","last_became_peered":"2024-09-16T02:14:22.482031+0000","last_unstale":"2024-09-16T02:14:34.358692+0000","last_undegraded":"2024-09-16T02:14:34.358692+0000","last_fullsized":"2024-09-16T02:14:34.358692+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:40.337516+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.015134246000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,15,4,14,11,2],"acting":[12,1,15,4,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.8","version":"45'23","reported_seq":101,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.348362+0000","last_change":"2024-09-16T02:14:34.348362+0000","last_active":"2024-09-16T02:14:34.348362+0000","last_peered":"2024-09-16T02:14:34.348362+0000","last_clean":"2024-09-16T02:14:34.348362+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:14:34.348362+0000","last_undegraded":"2024-09-16T02:14:34.348362+0000","last_fullsized":"2024-09-16T02:14:34.348362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.889399+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0079549649999999996,"stat_sum":{"num_bytes":950272,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":32,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"1.0","version":"26'112","reported_seq":186,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.355512+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:34.355512+0000","last_peered":"2024-09-16T02:14:34.355512+0000","last_clean":"2024-09-16T02:14:34.355512+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:34.355512+0000","last_undegraded":"2024-09-16T02:14:34.355512+0000","last_fullsized":"2024-09-16T02:14:34.355512+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":55,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.340441+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:34.340441+0000","last_peered":"2024-09-16T02:14:34.340441+0000","last_clean":"2024-09-16T02:14:34.340441+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:34.340441+0000","last_undegraded":"2024-09-16T02:14:34.340441+0000","last_fullsized":"2024-09-16T02:14:34.340441+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"45'32","reported_seq":121,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.963143+0000","last_change":"2024-09-16T02:14:34.964955+0000","last_active":"2024-09-16T02:14:34.963143+0000","last_peered":"2024-09-16T02:14:34.963143+0000","last_clean":"2024-09-16T02:14:34.963143+0000","last_became_active":"2024-09-16T02:14:22.476669+0000","last_became_peered":"2024-09-16T02:14:22.476669+0000","last_unstale":"2024-09-16T02:14:34.963143+0000","last_undegraded":"2024-09-16T02:14:34.963143+0000","last_fullsized":"2024-09-16T02:14:34.963143+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:45.354319+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.60913210200000001,"stat_sum":{"num_bytes":4423680,"num_objects":7,"num_object_clones":0,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":2520,"num_write":21,"num_write_kb":4896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,10,14,1,0],"acting":[13,12,6,10,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.0","version":"0'0","reported_seq":53,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.347873+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:34.347873+0000","last_peered":"2024-09-16T02:14:34.347873+0000","last_clean":"2024-09-16T02:14:34.347873+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:34.347873+0000","last_undegraded":"2024-09-16T02:14:34.347873+0000","last_fullsized":"2024-09-16T02:14:34.347873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"45'49","reported_seq":123,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.847907+0000","last_change":"2024-09-16T02:14:34.849965+0000","last_active":"2024-09-16T02:14:34.847907+0000","last_peered":"2024-09-16T02:14:34.847907+0000","last_clean":"2024-09-16T02:14:34.847907+0000","last_became_active":"2024-09-16T02:14:22.493319+0000","last_became_peered":"2024-09-16T02:14:22.493319+0000","last_unstale":"2024-09-16T02:14:34.847907+0000","last_undegraded":"2024-09-16T02:14:34.847907+0000","last_fullsized":"2024-09-16T02:14:34.847907+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:33.079673+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.49410925,"stat_sum":{"num_bytes":5521408,"num_objects":7,"num_object_clones":0,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":18,"num_read_kb":9,"num_write":45,"num_write_kb":6240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.355531+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:34.355531+0000","last_peered":"2024-09-16T02:14:34.355531+0000","last_clean":"2024-09-16T02:14:34.355531+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:34.355531+0000","last_undegraded":"2024-09-16T02:14:34.355531+0000","last_fullsized":"2024-09-16T02:14:34.355531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"45'3","reported_seq":75,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.348993+0000","last_change":"2024-09-16T02:14:34.348993+0000","last_active":"2024-09-16T02:14:34.348993+0000","last_peered":"2024-09-16T02:14:34.348993+0000","last_clean":"2024-09-16T02:14:34.348993+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:14:34.348993+0000","last_undegraded":"2024-09-16T02:14:34.348993+0000","last_fullsized":"2024-09-16T02:14:34.348993+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.599266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026594560000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":3,"num_write_kb":256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.2","version":"28'2","reported_seq":55,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.346735+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:34.346735+0000","last_peered":"2024-09-16T02:14:34.346735+0000","last_clean":"2024-09-16T02:14:34.346735+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:34.346735+0000","last_undegraded":"2024-09-16T02:14:34.346735+0000","last_fullsized":"2024-09-16T02:14:34.346735+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"45'20","reported_seq":97,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:35.041682+0000","last_change":"2024-09-16T02:14:35.043931+0000","last_active":"2024-09-16T02:14:35.041682+0000","last_peered":"2024-09-16T02:14:35.041682+0000","last_clean":"2024-09-16T02:14:35.041682+0000","last_became_active":"2024-09-16T02:14:22.484621+0000","last_became_peered":"2024-09-16T02:14:22.484621+0000","last_unstale":"2024-09-16T02:14:35.041682+0000","last_undegraded":"2024-09-16T02:14:35.041682+0000","last_fullsized":"2024-09-16T02:14:35.041682+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.574796+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.68796455899999998,"stat_sum":{"num_bytes":983040,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":132,"num_write":8,"num_write_kb":1392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,10],"acting":[13,2,11,14,4,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"3.5","version":"45'16","reported_seq":95,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.676395+0000","last_change":"2024-09-16T02:14:34.383251+0000","last_active":"2024-09-16T02:14:34.676395+0000","last_peered":"2024-09-16T02:14:34.676395+0000","last_clean":"2024-09-16T02:14:34.676395+0000","last_became_active":"2024-09-16T02:14:22.476388+0000","last_became_peered":"2024-09-16T02:14:22.476388+0000","last_unstale":"2024-09-16T02:14:34.676395+0000","last_undegraded":"2024-09-16T02:14:34.676395+0000","last_fullsized":"2024-09-16T02:14:34.676395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.082874+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.019771357999999999,"stat_sum":{"num_bytes":1654784,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":1864,"num_write":13,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,15,14,0,7,8],"acting":[10,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.354600+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:34.354600+0000","last_peered":"2024-09-16T02:14:34.354600+0000","last_clean":"2024-09-16T02:14:34.354600+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:34.354600+0000","last_undegraded":"2024-09-16T02:14:34.354600+0000","last_fullsized":"2024-09-16T02:14:34.354600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"45'10","reported_seq":82,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.361572+0000","last_change":"2024-09-16T02:14:34.363286+0000","last_active":"2024-09-16T02:14:34.361572+0000","last_peered":"2024-09-16T02:14:34.361572+0000","last_clean":"2024-09-16T02:14:34.361572+0000","last_became_active":"2024-09-16T02:14:22.346842+0000","last_became_peered":"2024-09-16T02:14:22.346842+0000","last_unstale":"2024-09-16T02:14:34.361572+0000","last_undegraded":"2024-09-16T02:14:34.361572+0000","last_fullsized":"2024-09-16T02:14:34.361572+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:40.854539+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016526658999999999,"stat_sum":{"num_bytes":1458176,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":7,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":50,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.353186+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:33.353186+0000","last_peered":"2024-09-16T02:14:33.353186+0000","last_clean":"2024-09-16T02:14:33.353186+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:33.353186+0000","last_undegraded":"2024-09-16T02:14:33.353186+0000","last_fullsized":"2024-09-16T02:14:33.353186+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"45'32","reported_seq":106,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.374113+0000","last_change":"2024-09-16T02:14:34.376398+0000","last_active":"2024-09-16T02:14:34.374113+0000","last_peered":"2024-09-16T02:14:34.374113+0000","last_clean":"2024-09-16T02:14:34.374113+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:14:34.374113+0000","last_undegraded":"2024-09-16T02:14:34.374113+0000","last_fullsized":"2024-09-16T02:14:34.374113+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.741942+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.024490867999999999,"stat_sum":{"num_bytes":2654208,"num_objects":5,"num_object_clones":0,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":6,"num_write":17,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"2.6","version":"0'0","reported_seq":52,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.336952+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:33.336952+0000","last_peered":"2024-09-16T02:14:33.336952+0000","last_clean":"2024-09-16T02:14:33.336952+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:33.336952+0000","last_undegraded":"2024-09-16T02:14:33.336952+0000","last_fullsized":"2024-09-16T02:14:33.336952+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"35'4","reported_seq":71,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-16T02:14:33.337164+0000","last_change":"2024-09-16T02:14:33.337534+0000","last_active":"2024-09-16T02:14:33.337164+0000","last_peered":"2024-09-16T02:14:33.337164+0000","last_clean":"2024-09-16T02:14:33.337164+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:14:33.337164+0000","last_undegraded":"2024-09-16T02:14:33.337164+0000","last_fullsized":"2024-09-16T02:14:33.337164+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:46.302344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080905199999999999,"stat_sum":{"num_bytes":393216,"num_objects":2,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":4,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-16T02:14:34.341699+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:34.341699+0000","last_peered":"2024-09-16T02:14:34.341699+0000","last_clean":"2024-09-16T02:14:34.341699+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:34.341699+0000","last_undegraded":"2024-09-16T02:14:34.341699+0000","last_fullsized":"2024-09-16T02:14:34.341699+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":24133632,"num_objects":46,"num_object_clones":2,"num_object_copies":322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":154,"num_read_kb":6829,"num_write":187,"num_write_kb":30496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":43831296,"data_stored":45985792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34304,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337536,"statfs":{"total":96636764160,"available":96601636864,"internally_reserved":0,"allocated":3186688,"data_stored":3734912,"data_compressed":99322,"data_compressed_allocated":135168,"data_compressed_original":700416,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29828,"kb_used_data":2792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342012,"statfs":{"total":96636764160,"available":96606220288,"internally_reserved":0,"allocated":2859008,"data_stored":2931552,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34416,"kb_used_data":3228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337424,"statfs":{"total":96636764160,"available":96601522176,"internally_reserved":0,"allocated":3305472,"data_stored":3169120,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313226,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31948,"kb_used_data":4912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339892,"statfs":{"total":96636764160,"available":96604049408,"internally_reserved":0,"allocated":5029888,"data_stored":5817119,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30828,"kb_used_data":3796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341012,"statfs":{"total":96636764160,"available":96605196288,"internally_reserved":0,"allocated":3887104,"data_stored":4309791,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313224,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31176,"kb_used_data":4140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340664,"statfs":{"total":96636764160,"available":96604839936,"internally_reserved":0,"allocated":4239360,"data_stored":4866847,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280521,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29916,"kb_used_data":2888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341924,"statfs":{"total":96636764160,"available":96606130176,"internally_reserved":0,"allocated":2957312,"data_stored":3154738,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30076,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341764,"statfs":{"total":96636764160,"available":96605966336,"internally_reserved":0,"allocated":3121152,"data_stored":2939744,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280520,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30140,"kb_used_data":3112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341700,"statfs":{"total":96636764160,"available":96605900800,"internally_reserved":0,"allocated":3186688,"data_stored":2993011,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280521,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29464,"kb_used_data":2432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342376,"statfs":{"total":96636764160,"available":96606593024,"internally_reserved":0,"allocated":2490368,"data_stored":2667295,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":5096,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":5218304,"data_stored":6731071,"data_compressed":96357,"data_compressed_allocated":131072,"data_compressed_original":692224,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30860,"kb_used_data":3824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340980,"statfs":{"total":96636764160,"available":96605163520,"internally_reserved":0,"allocated":3915776,"data_stored":3771232,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28456,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343384,"statfs":{"total":96636764160,"available":96607625216,"internally_reserved":0,"allocated":1454080,"data_stored":1272672,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280521,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35344,"kb_used_data":4156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336496,"statfs":{"total":96636764160,"available":96600571904,"internally_reserved":0,"allocated":4255744,"data_stored":4373344,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280521,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30540,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341300,"statfs":{"total":96636764160,"available":96605491200,"internally_reserved":0,"allocated":3592192,"data_stored":4199199,"data_compressed":63826,"data_compressed_allocated":90112,"data_compressed_original":180224,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280521,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30080,"kb_used_data":3052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341760,"statfs":{"total":96636764160,"available":96605962240,"internally_reserved":0,"allocated":3125248,"data_stored":2943840,"data_compressed":66791,"data_compressed_allocated":94208,"data_compressed_original":188416,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4022272,"data_stored":4022272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2572288,"data_stored":2572288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1634304,"data_stored":1634304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3215360,"data_stored":3215360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":860160,"data_stored":860160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2527232,"data_stored":2527232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3407872,"data_stored":3407872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2699264,"data_stored":2699264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2019328,"data_stored":2383872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2908160,"data_stored":2908160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2531328,"data_stored":2531328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2494464,"data_stored":2494464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":2228224,"data_stored":2228224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":2383872,"data_stored":2383872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":4558848,"data_stored":5533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":3768320,"data_stored":4583424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:35.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-09-16T02:14:35.611 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:35.612 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:35.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:35.614+0000 7f37a7fff640 0 -- 172.21.15.3:0/2027820492 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f37a8224830 msgr2=0x7f37a8226c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:35.619 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:35.618+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:35.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:35.645 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:35.720 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:35.720+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:35.721 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:35.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:14:35.821 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:35.818+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:35.868 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:35.868 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:35.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:14:35.922 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:35.920+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:36.023 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:36.022+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:36.044 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.044 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.059 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:14:36.104 INFO:tasks.rados.rados.0.smithi195.stdout:103: finishing rollback tid 2 to smithi19530025-18 2024-09-16T02:14:36.105 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 18 v 8 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-16T02:14:36.124 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:36.120+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:36.160 INFO:tasks.rados.rados.0.smithi195.stdout:102: finishing copy_from to smithi19530025-14 2024-09-16T02:14:36.160 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 14 v 2 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-16T02:14:36.176 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.176 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.190 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:14:36.225 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:36.218+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:36.315 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.315 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.326 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:36.322+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:36.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:14:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 3 pgp_num to 16 2024-09-16T02:14:36.356 INFO:tasks.rados.rados.0.smithi195.stdout:101: done (3 left) 2024-09-16T02:14:36.356 INFO:tasks.rados.rados.0.smithi195.stdout:102: done (2 left) 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:103: done (1 left) 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:104: done (0 left) 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:105: read oid 34 snap -1 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:105: expect (ObjNum 58 snap 5 seq_num 58) 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:106: rollback oid 16 current snap is 7 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 16 to 5 2024-09-16T02:14:36.357 INFO:tasks.rados.rados.0.smithi195.stdout:107: delete oid 46 current snap is 7 2024-09-16T02:14:36.370 INFO:tasks.rados.rados.0.smithi195.stdout:105: expect (ObjNum 58 snap 5 seq_num 58) 2024-09-16T02:14:36.370 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:36.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:14:36.416 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:36.427 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:36.426+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:36.447 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:36.447 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:36.456 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.456 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.458 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1468150 2024-09-16T02:14:36.470 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:14:36.528 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:36.523+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:36.626 INFO:tasks.rados.rados.0.smithi195.stdout:106: finishing rollback tid 1 to smithi19530025-16 2024-09-16T02:14:36.626 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 16 v 0 (ObjNum 129 snap 0 seq_num 3151132984) dirty dne 2024-09-16T02:14:36.628 INFO:tasks.rados.rados.0.smithi195.stdout:105: done (2 left) 2024-09-16T02:14:36.628 INFO:tasks.rados.rados.0.smithi195.stdout:106: done (1 left) 2024-09-16T02:14:36.628 INFO:tasks.rados.rados.0.smithi195.stdout:107: done (0 left) 2024-09-16T02:14:36.628 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 16 current snap is 7 2024-09-16T02:14:36.628 INFO:tasks.rados.rados.0.smithi195.stdout:108: seq_num 64 ranges {0=704512,704512=376832} 2024-09-16T02:14:36.629 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:36.622+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:36.637 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.637 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.640 INFO:tasks.rados.rados.0.smithi195.stdout:108: writing smithi19530025-16 from 0 to 704512 tid 1 2024-09-16T02:14:36.647 INFO:tasks.rados.rados.0.smithi195.stdout:108: writing smithi19530025-16 from 704512 to 1081344 tid 2 2024-09-16T02:14:36.647 INFO:tasks.rados.rados.0.smithi195.stdout:109: read oid 48 snap -1 2024-09-16T02:14:36.647 INFO:tasks.rados.rados.0.smithi195.stdout:109: expect (ObjNum 59 snap 5 seq_num 59) 2024-09-16T02:14:36.647 INFO:tasks.rados.rados.0.smithi195.stdout:110: snap_remove snap 5 2024-09-16T02:14:36.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:14:36.729 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:36.722+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:36.770 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.770 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:14:36.830 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:36.826+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:36.904 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:36.904 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:36.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:14:36.932 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:36.930+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:37.033 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:37.030+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:37.040 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.040 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:14:37.138 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:37.138+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:37.212 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.212 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:14:37.234 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:37.231+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:37.335 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:37.334+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:37.366 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:110: done (2 left) 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:111: copy_from oid 34 from oid 8 current snap is 7 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: finishing write tid 1 to smithi19530025-16 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: finishing write tid 2 to smithi19530025-16 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: finishing write tid 3 to smithi19530025-16 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: finishing write tid 4 to smithi19530025-16 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: oid 16 updating version 0 to 6 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: oid 16 updating version 6 to 7 2024-09-16T02:14:37.374 INFO:tasks.rados.rados.0.smithi195.stdout:108: oid 16 version 7 is already newer than 5 2024-09-16T02:14:37.375 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 16 v 7 (ObjNum 64 snap 7 seq_num 64) dirty exists 2024-09-16T02:14:37.375 INFO:tasks.rados.rados.0.smithi195.stdout:108: left oid 16 (ObjNum 64 snap 7 seq_num 64) 2024-09-16T02:14:37.375 INFO:tasks.rados.rados.0.smithi195.stdout:109: expect (ObjNum 59 snap 5 seq_num 59) 2024-09-16T02:14:37.375 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.389 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.389 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.402 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:14:37.435 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1133311 2024-09-16T02:14:37.443 INFO:tasks.rados.rados.0.smithi195.stdout:111: finishing copy_from racing read to smithi19530025-34 2024-09-16T02:14:37.443 INFO:tasks.rados.rados.0.smithi195.stdout:111: finishing copy_from to smithi19530025-34 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 17 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:108: done (2 left) 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:109: done (1 left) 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:111: done (0 left) 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 35 current snap is 7 2024-09-16T02:14:37.444 INFO:tasks.rados.rados.0.smithi195.stdout:112: seq_num 65 ranges {0=720896,720896=376832} 2024-09-16T02:14:37.450 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:37.450+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:37.455 INFO:tasks.rados.rados.0.smithi195.stdout:112: writing smithi19530025-35 from 0 to 720896 tid 1 2024-09-16T02:14:37.463 INFO:tasks.rados.rados.0.smithi195.stdout:112: writing smithi19530025-35 from 720896 to 1097728 tid 2 2024-09-16T02:14:37.463 INFO:tasks.rados.rados.0.smithi195.stdout:113: delete oid 16 current snap is 7 2024-09-16T02:14:37.463 INFO:tasks.rados.rados.0.smithi195.stdout:112: finishing write tid 1 to smithi19530025-35 2024-09-16T02:14:37.471 INFO:tasks.rados.rados.0.smithi195.stdout:112: finishing write tid 2 to smithi19530025-35 2024-09-16T02:14:37.472 INFO:tasks.rados.rados.0.smithi195.stdout:112: finishing write tid 3 to smithi19530025-35 2024-09-16T02:14:37.479 INFO:tasks.rados.rados.0.smithi195.stdout:112: finishing write tid 4 to smithi19530025-35 2024-09-16T02:14:37.479 INFO:tasks.rados.rados.0.smithi195.stdout:112: oid 35 updating version 0 to 33 2024-09-16T02:14:37.480 INFO:tasks.rados.rados.0.smithi195.stdout:112: oid 35 updating version 33 to 34 2024-09-16T02:14:37.480 INFO:tasks.rados.rados.0.smithi195.stdout:112: oid 35 updating version 34 to 35 2024-09-16T02:14:37.480 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 35 v 35 (ObjNum 65 snap 7 seq_num 65) dirty exists 2024-09-16T02:14:37.480 INFO:tasks.rados.rados.0.smithi195.stdout:112: left oid 35 (ObjNum 65 snap 7 seq_num 65) 2024-09-16T02:14:37.529 INFO:tasks.rados.rados.0.smithi195.stdout:112: done (1 left) 2024-09-16T02:14:37.529 INFO:tasks.rados.rados.0.smithi195.stdout:113: done (0 left) 2024-09-16T02:14:37.529 INFO:tasks.rados.rados.0.smithi195.stdout:114: read oid 1 snap 6 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:114: expect (ObjNum 56 snap 5 seq_num 56) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:115: read oid 18 snap -1 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:115: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:116: rmattr oid 7 current snap is 7 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:116: done (2 left) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:117: rmattr oid 22 current snap is 7 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:117: done (2 left) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:118: read oid 3 snap -1 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:118: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:119: read oid 26 snap 6 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:119: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:120: read oid 12 snap -1 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:120: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:121: rollback oid 11 current snap is 7 2024-09-16T02:14:37.530 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 11 to 6 2024-09-16T02:14:37.531 INFO:tasks.rados.rados.0.smithi195.stdout:122: rollback oid 42 current snap is 7 2024-09-16T02:14:37.531 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 42 to 6 2024-09-16T02:14:37.531 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 37 current snap is 7 2024-09-16T02:14:37.531 INFO:tasks.rados.rados.0.smithi195.stdout:123: seq_num 67 ranges {606208=753664,1359872=376832} 2024-09-16T02:14:37.537 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:37.535+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:37.559 INFO:tasks.rados.rados.0.smithi195.stdout:123: writing smithi19530025-37 from 606208 to 1359872 tid 1 2024-09-16T02:14:37.566 INFO:tasks.rados.rados.0.smithi195.stdout:123: writing smithi19530025-37 from 1359872 to 1736704 tid 2 2024-09-16T02:14:37.566 INFO:tasks.rados.rados.0.smithi195.stdout:124: read oid 6 snap -1 2024-09-16T02:14:37.566 INFO:tasks.rados.rados.0.smithi195.stdout:124: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:14:37.567 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 46 current snap is 7 2024-09-16T02:14:37.567 INFO:tasks.rados.rados.0.smithi195.stdout:125: seq_num 68 ranges {0=770048,770048=376832} 2024-09-16T02:14:37.580 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.580 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.580 INFO:tasks.rados.rados.0.smithi195.stdout:125: writing smithi19530025-46 from 0 to 770048 tid 1 2024-09-16T02:14:37.586 INFO:tasks.rados.rados.0.smithi195.stdout:125: writing smithi19530025-46 from 770048 to 1146880 tid 2 2024-09-16T02:14:37.586 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 13 current snap is 7 2024-09-16T02:14:37.586 INFO:tasks.rados.rados.0.smithi195.stdout:126: seq_num 69 ranges {212992=786432,999424=376832} 2024-09-16T02:14:37.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:14:37.598 INFO:tasks.rados.rados.0.smithi195.stdout:126: writing smithi19530025-13 from 212992 to 999424 tid 1 2024-09-16T02:14:37.601 INFO:tasks.rados.rados.0.smithi195.stdout:126: writing smithi19530025-13 from 999424 to 1376256 tid 2 2024-09-16T02:14:37.602 INFO:tasks.rados.rados.0.smithi195.stdout:127: rollback oid 41 current snap is 7 2024-09-16T02:14:37.602 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 41 to 6 2024-09-16T02:14:37.602 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 48 current snap is 7 2024-09-16T02:14:37.602 INFO:tasks.rados.rados.0.smithi195.stdout:128: seq_num 70 ranges {1802240=802816,2605056=376832} 2024-09-16T02:14:37.623 INFO:tasks.rados.rados.0.smithi195.stdout:128: writing smithi19530025-48 from 1802240 to 2605056 tid 1 2024-09-16T02:14:37.626 INFO:tasks.rados.rados.0.smithi195.stdout:128: writing smithi19530025-48 from 2605056 to 2981888 tid 2 2024-09-16T02:14:37.626 INFO:tasks.rados.rados.0.smithi195.stdout:129: copy_from oid 49 from oid 43 current snap is 7 2024-09-16T02:14:37.626 INFO:tasks.rados.rados.0.smithi195.stdout:119: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T02:14:37.632 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 339083 2024-09-16T02:14:37.645 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:37.643+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:37.645 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.668 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 392691 2024-09-16T02:14:37.673 INFO:tasks.rados.rados.0.smithi195.stdout:118: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-16T02:14:37.673 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.684 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 21406 2024-09-16T02:14:37.696 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 414456 2024-09-16T02:14:37.699 INFO:tasks.rados.rados.0.smithi195.stdout:114: expect (ObjNum 56 snap 5 seq_num 56) 2024-09-16T02:14:37.703 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 535702 2024-09-16T02:14:37.708 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.719 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.726 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.727 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.729 INFO:tasks.rados.rados.0.smithi195.stdout:115: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-16T02:14:37.729 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.732 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.734 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:122: finishing rollback tid 0 to smithi19530025-42 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:122: finishing rollback tid 1 to smithi19530025-42 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:121: finishing rollback tid 0 to smithi19530025-11 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:121: finishing rollback tid 1 to smithi19530025-11 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:121: finishing rollback tid 2 to smithi19530025-11 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 57 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:120: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T02:14:37.737 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.738 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.739 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:37.731+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:37.741 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 57563 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:123: finishing write tid 1 to smithi19530025-37 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:123: finishing write tid 2 to smithi19530025-37 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:123: finishing write tid 3 to smithi19530025-37 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:124: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 35220 2024-09-16T02:14:37.742 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.743 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:123: finishing write tid 4 to smithi19530025-37 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:123: oid 37 updating version 0 to 24 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:123: oid 37 version 24 is already newer than 22 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:123: oid 37 version 24 is already newer than 23 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 24 (ObjNum 67 snap 7 seq_num 67) dirty exists 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:123: left oid 37 (ObjNum 67 snap 7 seq_num 67) 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: finishing write tid 1 to smithi19530025-46 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: finishing write tid 2 to smithi19530025-46 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: finishing write tid 3 to smithi19530025-46 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: finishing write tid 4 to smithi19530025-46 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: oid 46 updating version 0 to 27 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: oid 46 version 27 is already newer than 26 2024-09-16T02:14:37.745 INFO:tasks.rados.rados.0.smithi195.stdout:125: oid 46 version 27 is already newer than 25 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 27 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:125: left oid 46 (ObjNum 68 snap 7 seq_num 68) 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: finishing write tid 1 to smithi19530025-13 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: finishing write tid 2 to smithi19530025-13 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: finishing write tid 3 to smithi19530025-13 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: finishing write tid 4 to smithi19530025-13 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: oid 13 updating version 0 to 63 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: oid 13 updating version 63 to 65 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: oid 13 version 65 is already newer than 64 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 13 v 65 (ObjNum 69 snap 7 seq_num 69) dirty exists 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:126: left oid 13 (ObjNum 69 snap 7 seq_num 69) 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:127: finishing rollback tid 0 to smithi19530025-41 2024-09-16T02:14:37.746 INFO:tasks.rados.rados.0.smithi195.stdout:127: finishing rollback tid 1 to smithi19530025-41 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:122: finishing rollback tid 2 to smithi19530025-42 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 42 v 39 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:127: finishing rollback tid 2 to smithi19530025-41 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 41 v 69 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:128: finishing write tid 1 to smithi19530025-48 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:129: finishing copy_from racing read to smithi19530025-49 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:128: finishing write tid 2 to smithi19530025-48 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:114: done (13 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:115: done (12 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:118: done (11 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:119: done (10 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:120: done (9 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:121: done (8 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:122: done (7 left) 2024-09-16T02:14:37.747 INFO:tasks.rados.rados.0.smithi195.stdout:123: done (6 left) 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:124: done (5 left) 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:125: done (4 left) 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:126: done (3 left) 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:127: done (2 left) 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 50 current snap is 7 2024-09-16T02:14:37.748 INFO:tasks.rados.rados.0.smithi195.stdout:130: seq_num 71 ranges {835584=425984,1261568=573440,1835008=196608} 2024-09-16T02:14:37.765 INFO:tasks.rados.rados.0.smithi195.stdout:130: writing smithi19530025-50 from 835584 to 1261568 tid 1 2024-09-16T02:14:37.775 INFO:tasks.rados.rados.0.smithi195.stdout:130: writing smithi19530025-50 from 1261568 to 1835008 tid 2 2024-09-16T02:14:37.778 INFO:tasks.rados.rados.0.smithi195.stdout:130: writing smithi19530025-50 from 1835008 to 2031616 tid 3 2024-09-16T02:14:37.778 INFO:tasks.rados.rados.0.smithi195.stdout:131: read oid 20 snap -1 2024-09-16T02:14:37.778 INFO:tasks.rados.rados.0.smithi195.stdout:131: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-16T02:14:37.778 INFO:tasks.rados.rados.0.smithi195.stdout:132: setattr oid 21 current snap is 7 2024-09-16T02:14:37.785 INFO:tasks.rados.rados.0.smithi195.stdout:130: finishing write tid 1 to smithi19530025-50 2024-09-16T02:14:37.786 INFO:tasks.rados.rados.0.smithi195.stdout:130: finishing write tid 2 to smithi19530025-50 2024-09-16T02:14:37.787 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 2 current snap is 7 2024-09-16T02:14:37.787 INFO:tasks.rados.rados.0.smithi195.stdout:133: seq_num 72 ranges {32768=442368,475136=720896,1196032=49152} 2024-09-16T02:14:37.794 INFO:tasks.rados.rados.0.smithi195.stdout:133: writing smithi19530025-2 from 32768 to 475136 tid 1 2024-09-16T02:14:37.803 INFO:tasks.rados.rados.0.smithi195.stdout:133: writing smithi19530025-2 from 475136 to 1196032 tid 2 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:133: writing smithi19530025-2 from 1196032 to 1245184 tid 3 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:134: read oid 8 snap -1 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:134: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:135: read oid 1 snap 6 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:135: expect (ObjNum 56 snap 5 seq_num 56) 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:136: rollback oid 19 current snap is 7 2024-09-16T02:14:37.805 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 19 to 6 2024-09-16T02:14:37.806 INFO:tasks.rados.rados.0.smithi195.stdout:130: finishing write tid 3 to smithi19530025-50 2024-09-16T02:14:37.806 INFO:tasks.rados.rados.0.smithi195.stdout:131: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-16T02:14:37.806 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 55550 2024-09-16T02:14:37.810 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 260849 2024-09-16T02:14:37.810 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 21 v 12 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:133: finishing write tid 1 to smithi19530025-2 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:131: done (8 left) 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:132: done (7 left) 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 24 current snap is 7 2024-09-16T02:14:37.816 INFO:tasks.rados.rados.0.smithi195.stdout:137: seq_num 73 ranges {393216=458752,851968=475136,1327104=294912} 2024-09-16T02:14:37.823 INFO:tasks.rados.rados.0.smithi195.stdout:137: writing smithi19530025-24 from 393216 to 851968 tid 1 2024-09-16T02:14:37.825 INFO:tasks.rados.rados.0.smithi195.stdout:137: writing smithi19530025-24 from 851968 to 1327104 tid 2 2024-09-16T02:14:37.828 INFO:tasks.rados.rados.0.smithi195.stdout:137: writing smithi19530025-24 from 1327104 to 1622016 tid 3 2024-09-16T02:14:37.828 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 17 current snap is 7 2024-09-16T02:14:37.828 INFO:tasks.rados.rados.0.smithi195.stdout:138: seq_num 74 ranges {278528=475136,753664=622592,1376256=147456} 2024-09-16T02:14:37.834 INFO:tasks.rados.rados.0.smithi195.stdout:138: writing smithi19530025-17 from 278528 to 753664 tid 1 2024-09-16T02:14:37.838 INFO:tasks.rados.rados.0.smithi195.stdout:138: writing smithi19530025-17 from 753664 to 1376256 tid 2 2024-09-16T02:14:37.839 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:37.838+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:37.840 INFO:tasks.rados.rados.0.smithi195.stdout:138: writing smithi19530025-17 from 1376256 to 1523712 tid 3 2024-09-16T02:14:37.840 INFO:tasks.rados.rados.0.smithi195.stdout:139: read oid 47 snap -1 2024-09-16T02:14:37.840 INFO:tasks.rados.rados.0.smithi195.stdout:139: expect deleted 2024-09-16T02:14:37.840 INFO:tasks.rados.rados.0.smithi195.stdout:140: snap_create 2024-09-16T02:14:37.840 INFO:tasks.rados.rados.0.smithi195.stdout:133: finishing write tid 2 to smithi19530025-2 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: finishing write tid 3 to smithi19530025-2 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: finishing write tid 4 to smithi19530025-2 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: finishing write tid 5 to smithi19530025-2 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: oid 2 updating version 0 to 26 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: oid 2 updating version 26 to 28 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: oid 2 version 28 is already newer than 25 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: oid 2 version 28 is already newer than 27 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 2 v 28 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:133: left oid 2 (ObjNum 72 snap 7 seq_num 72) 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:136: finishing rollback tid 0 to smithi19530025-19 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:136: finishing rollback tid 1 to smithi19530025-19 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:136: finishing rollback tid 2 to smithi19530025-19 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 19 v 38 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-16T02:14:37.841 INFO:tasks.rados.rados.0.smithi195.stdout:135: expect (ObjNum 56 snap 5 seq_num 56) 2024-09-16T02:14:37.842 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.868 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 44936 2024-09-16T02:14:37.887 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:37.903 INFO:tasks.rados.rados.0.smithi195.stdout:137: finishing write tid 1 to smithi19530025-24 2024-09-16T02:14:37.903 INFO:tasks.rados.rados.0.smithi195.stdout:137: finishing write tid 2 to smithi19530025-24 2024-09-16T02:14:37.903 INFO:tasks.rados.rados.0.smithi195.stdout:137: finishing write tid 3 to smithi19530025-24 2024-09-16T02:14:37.903 INFO:tasks.rados.rados.0.smithi195.stdout:137: finishing write tid 4 to smithi19530025-24 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: finishing write tid 5 to smithi19530025-24 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: oid 24 updating version 0 to 21 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: oid 24 version 21 is already newer than 19 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: oid 24 version 21 is already newer than 20 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: oid 24 version 21 is already newer than 18 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 24 v 21 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:137: left oid 24 (ObjNum 73 snap 7 seq_num 73) 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: finishing write tid 1 to smithi19530025-17 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: finishing write tid 2 to smithi19530025-17 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: finishing write tid 3 to smithi19530025-17 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: finishing write tid 4 to smithi19530025-17 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: finishing write tid 5 to smithi19530025-17 2024-09-16T02:14:37.904 INFO:tasks.rados.rados.0.smithi195.stdout:138: oid 17 updating version 0 to 12 2024-09-16T02:14:37.905 INFO:tasks.rados.rados.0.smithi195.stdout:138: oid 17 version 12 is already newer than 11 2024-09-16T02:14:37.905 INFO:tasks.rados.rados.0.smithi195.stdout:138: oid 17 version 12 is already newer than 10 2024-09-16T02:14:37.905 INFO:tasks.rados.rados.0.smithi195.stdout:138: oid 17 updating version 12 to 13 2024-09-16T02:14:37.905 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 17 v 13 (ObjNum 74 snap 7 seq_num 74) dirty exists 2024-09-16T02:14:37.905 INFO:tasks.rados.rados.0.smithi195.stdout:138: left oid 17 (ObjNum 74 snap 7 seq_num 74) 2024-09-16T02:14:37.922 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:37.922 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:37.936 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:14:37.939 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:37.938+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:38.041 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:38.038+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:38.053 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.053 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.068 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:14:38.085 INFO:tasks.rados.rados.0.smithi195.stdout:128: finishing write tid 3 to smithi19530025-48 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:128: finishing write tid 4 to smithi19530025-48 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:128: oid 48 updating version 0 to 73 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:128: oid 48 version 73 is already newer than 71 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:128: oid 48 version 73 is already newer than 72 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 73 (ObjNum 70 snap 7 seq_num 70) dirty exists 2024-09-16T02:14:38.088 INFO:tasks.rados.rados.0.smithi195.stdout:128: left oid 48 (ObjNum 70 snap 7 seq_num 70) 2024-09-16T02:14:38.142 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:38.139+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:38.192 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.193 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.207 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:14:38.243 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:38.242+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:38.344 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:38.342+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:38.373 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.373 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:128: done (10 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:133: done (9 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:135: done (8 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:136: done (7 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:137: done (6 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:138: done (5 left) 2024-09-16T02:14:38.378 INFO:tasks.rados.rados.0.smithi195.stdout:139: done (4 left) 2024-09-16T02:14:38.379 INFO:tasks.rados.rados.0.smithi195.stdout:140: done (3 left) 2024-09-16T02:14:38.379 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 45 current snap is 8 2024-09-16T02:14:38.379 INFO:tasks.rados.rados.0.smithi195.stdout:141: seq_num 75 ranges {0=491520,491520=770048} 2024-09-16T02:14:38.383 INFO:tasks.rados.rados.0.smithi195.stdout:141: writing smithi19530025-45 from 0 to 491520 tid 1 2024-09-16T02:14:38.389 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:14:38.391 INFO:tasks.rados.rados.0.smithi195.stdout:141: writing smithi19530025-45 from 491520 to 1261568 tid 2 2024-09-16T02:14:38.391 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 11 current snap is 8 2024-09-16T02:14:38.391 INFO:tasks.rados.rados.0.smithi195.stdout:142: seq_num 76 ranges {540672=507904,1048576=507904,1556480=262144} 2024-09-16T02:14:38.401 INFO:tasks.rados.rados.0.smithi195.stdout:142: writing smithi19530025-11 from 540672 to 1048576 tid 1 2024-09-16T02:14:38.407 INFO:tasks.rados.rados.0.smithi195.stdout:142: writing smithi19530025-11 from 1048576 to 1556480 tid 2 2024-09-16T02:14:38.410 INFO:tasks.rados.rados.0.smithi195.stdout:142: writing smithi19530025-11 from 1556480 to 1818624 tid 3 2024-09-16T02:14:38.410 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 10 current snap is 8 2024-09-16T02:14:38.411 INFO:tasks.rados.rados.0.smithi195.stdout:143: seq_num 77 ranges {163840=524288,688128=524288,1212416=245760} 2024-09-16T02:14:38.418 INFO:tasks.rados.rados.0.smithi195.stdout:143: writing smithi19530025-10 from 163840 to 688128 tid 1 2024-09-16T02:14:38.424 INFO:tasks.rados.rados.0.smithi195.stdout:143: writing smithi19530025-10 from 688128 to 1212416 tid 2 2024-09-16T02:14:38.428 INFO:tasks.rados.rados.0.smithi195.stdout:143: writing smithi19530025-10 from 1212416 to 1458176 tid 3 2024-09-16T02:14:38.428 INFO:tasks.rados.rados.0.smithi195.stdout:144: read oid 39 snap -1 2024-09-16T02:14:38.428 INFO:tasks.rados.rados.0.smithi195.stdout:144: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T02:14:38.428 INFO:tasks.rados.rados.0.smithi195.stdout:145: rmattr oid 1 current snap is 8 2024-09-16T02:14:38.428 INFO:tasks.rados.rados.0.smithi195.stdout:145: done (7 left) 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:146: rmattr oid 32 current snap is 8 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:146: done (7 left) 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:141: finishing write tid 1 to smithi19530025-45 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:141: finishing write tid 2 to smithi19530025-45 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:141: finishing write tid 3 to smithi19530025-45 2024-09-16T02:14:38.429 INFO:tasks.rados.rados.0.smithi195.stdout:141: finishing write tid 4 to smithi19530025-45 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:141: oid 45 updating version 0 to 31 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:141: oid 45 version 31 is already newer than 30 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:141: oid 45 version 31 is already newer than 29 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 31 (ObjNum 75 snap 8 seq_num 75) dirty exists 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:141: left oid 45 (ObjNum 75 snap 8 seq_num 75) 2024-09-16T02:14:38.430 INFO:tasks.rados.rados.0.smithi195.stdout:129: finishing copy_from to smithi19530025-49 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 49 v 10 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: finishing write tid 1 to smithi19530025-11 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: finishing write tid 2 to smithi19530025-11 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: finishing write tid 3 to smithi19530025-11 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: finishing write tid 4 to smithi19530025-11 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:147: read oid 40 snap 7 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:147: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:129: done (7 left) 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:141: done (6 left) 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:148: delete oid 18 current snap is 8 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: finishing write tid 5 to smithi19530025-11 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: oid 11 updating version 0 to 76 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: oid 11 updating version 76 to 77 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: oid 11 updating version 77 to 78 2024-09-16T02:14:38.431 INFO:tasks.rados.rados.0.smithi195.stdout:142: oid 11 version 78 is already newer than 75 2024-09-16T02:14:38.432 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 78 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-09-16T02:14:38.432 INFO:tasks.rados.rados.0.smithi195.stdout:142: left oid 11 (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:14:38.439 INFO:tasks.rados.rados.0.smithi195.stdout:144: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-16T02:14:38.439 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.445 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:38.439+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:38.453 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:38.468 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.481 INFO:tasks.rados.rados.0.smithi195.stdout:147: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T02:14:38.483 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 182313 2024-09-16T02:14:38.498 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 503490 2024-09-16T02:14:38.501 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:143: finishing write tid 1 to smithi19530025-10 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:143: finishing write tid 2 to smithi19530025-10 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:142: done (6 left) 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:144: done (5 left) 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:147: done (4 left) 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:148: done (3 left) 2024-09-16T02:14:38.511 INFO:tasks.rados.rados.0.smithi195.stdout:149: delete oid 46 current snap is 8 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: finishing write tid 3 to smithi19530025-10 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: finishing write tid 4 to smithi19530025-10 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: finishing write tid 5 to smithi19530025-10 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: oid 10 updating version 0 to 40 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: oid 10 updating version 40 to 43 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: oid 10 version 43 is already newer than 41 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: oid 10 version 43 is already newer than 42 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 10 v 43 (ObjNum 77 snap 8 seq_num 77) dirty exists 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:143: left oid 10 (ObjNum 77 snap 8 seq_num 77) 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:134: expect (ObjNum 61 snap 5 seq_num 61) 2024-09-16T02:14:38.512 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.520 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.520 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.527 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.535 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:14:38.542 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: finishing write tid 4 to smithi19530025-50 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: finishing write tid 5 to smithi19530025-50 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: oid 50 updating version 0 to 15 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: oid 50 updating version 15 to 17 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: oid 50 updating version 17 to 18 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: oid 50 version 18 is already newer than 16 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 50 v 18 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-09-16T02:14:38.557 INFO:tasks.rados.rados.0.smithi195.stdout:130: left oid 50 (ObjNum 71 snap 7 seq_num 71) 2024-09-16T02:14:38.558 INFO:tasks.rados.rados.0.smithi195.stdout:130: done (3 left) 2024-09-16T02:14:38.558 INFO:tasks.rados.rados.0.smithi195.stdout:134: done (2 left) 2024-09-16T02:14:38.558 INFO:tasks.rados.rados.0.smithi195.stdout:143: done (1 left) 2024-09-16T02:14:38.558 INFO:tasks.rados.rados.0.smithi195.stdout:149: done (0 left) 2024-09-16T02:14:38.558 INFO:tasks.rados.rados.0.smithi195.stdout:150: setattr oid 46 current snap is 8 2024-09-16T02:14:38.561 INFO:tasks.rados.rados.0.smithi195.stdout:151: snap_create 2024-09-16T02:14:38.565 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 30 (ObjNum -2134168352 snap 32550 seq_num 304) dirty exists 2024-09-16T02:14:38.581 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:38.579+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:38.641 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:38.641 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:38.646 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:38.639+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:38.657 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.657 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.667 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:38.667 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:38.671 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:14:38.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:38.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:38.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:38.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:38.738 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:38.757 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:38.755+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:38.763 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:38.763 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:38.833 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:38.833 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:38.847 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:14:38.848 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:38.843+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:38.886 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:38.886 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:38.924 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:38.924 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:38.948 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:38.942+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:38.963 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:38.963 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:39.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:39.003 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:39.010 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:39.010 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:39.024 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:14:39.041 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:39.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:39.048 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:39.042+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:39.079 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:39.080 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:39.117 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:39.117 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:39.140 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:39.140 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:39.150 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:39.150+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:39.154 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:14:39.156 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:39.156 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:39.158+0000 7fe6d3fff640 0 -- 172.21.15.3:0/803221074 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fe6dc2222c0 msgr2=0x7fe6dc2246b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:39.201 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:39.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:39.238 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:39.250 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:39.247+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:39.278 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:39.278 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:39.278 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:39.278 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:39.278 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:39.292 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:14:39.381 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:39.379+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:39.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:14:39.396 INFO:tasks.rados.rados.0.smithi195.stdout:150: done (1 left) 2024-09-16T02:14:39.397 INFO:tasks.rados.rados.0.smithi195.stdout:151: done (0 left) 2024-09-16T02:14:39.397 INFO:tasks.rados.rados.0.smithi195.stdout:152: snap_remove snap 7 2024-09-16T02:14:39.453 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:39.450+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=114, reqid=client.4598.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.16780354, 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014358203, 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.464 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294123+0000", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157742 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294233+0000", 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109842 2024-09-16T02:14:39.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294256+0000", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3779000000000001e-05 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294602+0000", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000345817 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294625+0000", 2024-09-16T02:14:39.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308228+0000", 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013602725 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308323+0000", 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4631999999999998e-05 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.468 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=115, reqid=client.4598.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.164027668999999, 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010657378, 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:39.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297742+0000", 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297835+0000", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2315e-05 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298241+0000", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040658699999999997 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298273+0000", 2024-09-16T02:14:39.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1566000000000002e-05 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298698+0000", 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042513000000000002 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298724+0000", 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5681999999999999e-05 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308331+0000", 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096073479999999999 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308398+0000", 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6749999999999996e-05 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.163867273999999, 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013936119, 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7299999999999996e-07 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298091+0000", 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000189612 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298361+0000", 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026983699999999998 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298377+0000", 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6619000000000001e-05 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298580+0000", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020297699999999999 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298602+0000", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1823000000000001e-05 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311758+0000", 2024-09-16T02:14:39.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013155635000000001 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311837+0000", 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9042999999999995e-05 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:39.478 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.163670165999999, 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013796841000000001, 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:39.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298099+0000", 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2999999999999999e-07 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298132+0000", 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3460999999999997e-05 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298437+0000", 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000305149 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298455+0000", 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8099000000000001e-05 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299840+0000", 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001384826 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5151999999999999e-05 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311844+0000", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011978284000000001 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311895+0000", 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1039999999999999e-05 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.482 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.5s3 35/32 ECSubWrite(tid=2, reqid=client.4598.0:13, at_version=35'3, trim_to=0'0, roll_forward_to=34'1))", 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.1628829, 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011300899, 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298887+0000", 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5400000000000001e-06 2024-09-16T02:14:39.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298992+0000", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105472 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299113+0000", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012086099999999999 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299135+0000", 2024-09-16T02:14:39.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1679e-05 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299294+0000", 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015895199999999999 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299322+0000", 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8246e-05 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310061+0000", 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010739165 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310186+0000", 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012498399999999999 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.487 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.160591956999999, 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010788813, 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301178+0000", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.77e-06 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301420+0000", 2024-09-16T02:14:39.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024188200000000001 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301595+0000", 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017550199999999999 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301613+0000", 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7391999999999999e-05 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302202+0000", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000588903 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302225+0000", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311905+0000", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096801509999999997 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311965+0000", 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9593999999999999e-05 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s3 35/32 ECSubWrite(tid=116, reqid=client.4598.0:2, at_version=0'0, trim_to=0'0, roll_forward_to=34'2))", 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301243+0000", 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.160524927000001, 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091685959999999993, 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.492 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301243+0000", 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301243+0000", 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301244+0000", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3399999999999997e-07 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301270+0000", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6271999999999999e-05 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301495+0000", 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000225116 2024-09-16T02:14:39.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301520+0000", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4151999999999998e-05 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301704+0000", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000184668 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301767+0000", 2024-09-16T02:14:39.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2626999999999995e-05 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310102+0000", 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083346840000000002 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310412+0000", 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031014299999999997 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.496 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.160341495999999, 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013049734, 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-09-16T02:14:39.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6052000000000001e-05 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000240423 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301716+0000", 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2289e-05 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305459+0000", 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037425710000000001 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305488+0000", 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8756e-05 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314378+0000", 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088900869999999996 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314476+0000", 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8669e-05 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.501 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.15995536, 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014210883000000001, 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301814+0000", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4130000000000001e-06 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302022+0000", 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000207844 2024-09-16T02:14:39.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302120+0000", 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7864999999999999e-05 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302140+0000", 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0156000000000001e-05 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310588+0000", 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084482120000000001 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310612+0000", 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3879e-05 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315921+0000", 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053087079999999997 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316024+0000", 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102806 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.158943981, 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013263980999999999, 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.506 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302826+0000", 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9240000000000001e-06 2024-09-16T02:14:39.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302901+0000", 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5172999999999993e-05 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304024+0000", 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001123181 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304072+0000", 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8127999999999997e-05 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312045+0000", 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079722669999999999 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312071+0000", 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6291999999999998e-05 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316033+0000", 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039623380000000001 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316088+0000", 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4678000000000001e-05 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.510 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.0s1 35/32 ECSubWrite(tid=0, reqid=client.4598.0:46, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.304398+0000", 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.157370327000001, 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010045073, 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304398+0000", 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304398+0000", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304398+0000", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7700000000000005e-07 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304780+0000", 2024-09-16T02:14:39.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038153800000000001 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304866+0000", 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6632999999999995e-05 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304898+0000", 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1662000000000003e-05 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305477+0000", 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057902599999999995 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305498+0000", 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0818999999999998e-05 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314139+0000", 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086412160000000002 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314443+0000", 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030370199999999999 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:49 3.cs0 3:3203a6be:::smithi19530025-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.145995949, 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016954778, 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.515 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315781+0000", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8699999999999998e-06 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316702+0000", 2024-09-16T02:14:39.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092113000000000004 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316734+0000", 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2375000000000001e-05 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316756+0000", 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1843000000000001e-05 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317070+0000", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031399899999999997 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317385+0000", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031517099999999999 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.320926+0000", 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035407450000000001 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332340+0000", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011414260000000001 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332532+0000", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191137 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332727+0000", 2024-09-16T02:14:39.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000195248 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:50 3.cs0 3:3203a6be:::smithi19530025-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.144955166999999, 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017275975999999998, 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:39.521 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316815+0000", 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0899999999999999e-06 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316826+0000", 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1242e-05 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316851+0000", 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4969000000000001e-05 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316868+0000", 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6535999999999999e-05 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324308+0000", 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074406289999999998 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324542+0000", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023414200000000001 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.325642+0000", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001099182 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332739+0000", 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070972090000000002 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.333048+0000", 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030949200000000002 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334089+0000", 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010404850000000001 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.526 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:51 3.cs0 3:3203a6be:::smithi19530025-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.144869671, 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019761212, 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 51 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316900+0000", 2024-09-16T02:14:39.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.477e-06 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316907+0000", 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8889999999999996e-06 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316927+0000", 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0591999999999999e-05 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316943+0000", 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5756999999999998e-05 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327066+0000", 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010123037 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327269+0000", 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309000000000001 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334254+0000", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069844 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334498+0000", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024477499999999997 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336660+0000", 2024-09-16T02:14:39.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021611949999999999 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=8, reqid=client.4598.0:41, at_version=0'0, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.317914+0000", 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.143853972, 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091426960000000005, 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317914+0000", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317914+0000", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317916+0000", 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8369999999999999e-06 2024-09-16T02:14:39.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318041+0000", 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000125284 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318143+0000", 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102169 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318166+0000", 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3112000000000002e-05 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318319+0000", 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015244600000000001 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318336+0000", 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7402999999999999e-05 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327000+0000", 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086634280000000008 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327057+0000", 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7017e-05 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=10, reqid=client.4598.0:64, at_version=35'7, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:39.536 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.323188+0000", 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.138579654999999, 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095239839999999992, 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323188+0000", 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323188+0000", 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323190+0000", 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9700000000000002e-06 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323339+0000", 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000148496 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323432+0000", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3098999999999998e-05 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323454+0000", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1974999999999999e-05 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323598+0000", 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000144017 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.323622+0000", 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3719000000000001e-05 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332640+0000", 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090185030000000006 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332712+0000", 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2205000000000004e-05 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=122, reqid=client.4598.0:58, at_version=35'5, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.326221+0000", 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.135546797, 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094582369999999995, 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.541 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.326221+0000", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.326221+0000", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.326223+0000", 2024-09-16T02:14:39.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0729999999999999e-06 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327343+0000", 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0011198110000000001 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327653+0000", 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031002900000000002 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327676+0000", 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2691999999999999e-05 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.329071+0000", 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013951689999999999 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.329094+0000", 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2929e-05 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.335575+0000", 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0064814240000000004 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.335679+0000", 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010411 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.545 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.7s6 44/32 ECSubWrite(tid=31, reqid=osd.3.0:31, at_version=44'28, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.348656+0000", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1131119209999998, 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094434589999999995, 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348656+0000", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348656+0000", 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348658+0000", 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7659999999999999e-06 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348720+0000", 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2391000000000006e-05 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348825+0000", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010454 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.348848+0000", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3002000000000001e-05 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349899+0000", 2024-09-16T02:14:39.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010511909999999999 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349924+0000", 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4958999999999999e-05 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.357990+0000", 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0080656040000000005 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.358100+0000", 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000110006 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.362537+0000", 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0992313579999999, 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089536330000000008, 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362537+0000", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362537+0000", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362537+0000", 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362583+0000", 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5581000000000003e-05 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364731+0000", 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021476070000000002 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364750+0000", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9758999999999998e-05 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364917+0000", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016625099999999999 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364940+0000", 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3485999999999998e-05 2024-09-16T02:14:39.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371399+0000", 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0064585420000000003 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371490+0000", 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1737000000000006e-05 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s3 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.365405+0000", 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0963627110000003, 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.008829735, 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365405+0000", 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365405+0000", 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365407+0000", 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1629999999999999e-06 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365451+0000", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4988999999999999e-05 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.366062+0000", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00061016100000000001 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.366081+0000", 2024-09-16T02:14:39.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9332999999999999e-05 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.367072+0000", 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00099064500000000007 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.367098+0000", 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6395000000000001e-05 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:39.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.374174+0000", 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.007075904 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.374235+0000", 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1144999999999998e-05 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:39.559 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:39.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:14:39.561 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:39.558+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:39.662 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:39.655+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:39.694 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:39.694 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:39.694 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:39.694 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:39.709 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:14:39.795 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:39.791+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:39.864 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:39.862+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:39.877 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:39.877 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:39.877 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:39.877 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:39.877 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:39.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:14:39.965 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:39.962+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:40.063 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=14, reqid=client.4598.0:197, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.326297+0000", 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.734391032, 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050539269999999997, 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326297+0000", 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326297+0000", 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326299+0000", 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326345+0000", 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5892e-05 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326440+0000", 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5268999999999999e-05 2024-09-16T02:14:40.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326463+0000", 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2972e-05 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327568+0000", 2024-09-16T02:14:40.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001105307 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327595+0000", 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6346000000000002e-05 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331296+0000", 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037013039999999999 2024-09-16T02:14:40.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331351+0000", 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4873999999999997e-05 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.724816596, 2024-09-16T02:14:40.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.334730205, 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:40.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335879+0000", 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335907+0000", 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7291999999999999e-05 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669030+0000", 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33312349400000002 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669045+0000", 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4943000000000001e-05 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669245+0000", 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019942899999999999 2024-09-16T02:14:40.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669263+0000", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8445999999999999e-05 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670533+0000", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001270024 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670602+0000", 2024-09-16T02:14:40.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8406999999999997e-05 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.348867+0000", 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.711820549, 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042234489999999998, 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.348867+0000", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.348867+0000", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.348868+0000", 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6599999999999996e-07 2024-09-16T02:14:40.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.349052+0000", 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018444300000000001 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.349167+0000", 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115022 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.349192+0000", 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4445999999999999e-05 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.349720+0000", 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052773299999999996 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.349750+0000", 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0134999999999999e-05 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.353003+0000", 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032528710000000001 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.353091+0000", 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8232999999999994e-05 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.7258437899999999, 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75881597999999995, 2024-09-16T02:14:40.079 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:40.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334845+0000", 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0900000000000001e-07 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334886+0000", 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1254999999999997e-05 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091945+0000", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75705888700000001 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091960+0000", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4686e-05 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092157+0000", 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019717000000000001 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092176+0000", 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9020000000000001e-05 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093578+0000", 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001401769 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093660+0000", 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2484000000000005e-05 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 45/32 ECSubWrite(tid=22, reqid=osd.6.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7093522050000001, 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061315420000000002, 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.084 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351337+0000", 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.773e-06 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351366+0000", 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8408999999999999e-05 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351446+0000", 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0513999999999995e-05 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351467+0000", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0913000000000001e-05 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352256+0000", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00078906799999999997 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352279+0000", 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2186000000000001e-05 2024-09-16T02:14:40.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357119+0000", 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048408469999999997 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357467+0000", 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034783200000000001 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.088 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7088206919999998, 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017058792, 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351868+0000", 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351917+0000", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352051+0000", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013457599999999999 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352073+0000", 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1489e-05 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352231+0000", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158455 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352260+0000", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8472000000000001e-05 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368854+0000", 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016594647000000001 2024-09-16T02:14:40.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368926+0000", 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1534999999999999e-05 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs5 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:40.093 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7027855599999997, 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088459079999999995, 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:40.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357905+0000", 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3599999999999999e-06 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357997+0000", 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2616999999999998e-05 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358121+0000", 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012370600000000001 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358231+0000", 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109963 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358421+0000", 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019028099999999999 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358634+0000", 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021316999999999999 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366661+0000", 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080266509999999992 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366748+0000", 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7159999999999999e-05 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7019484440000001, 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092364969999999998, 2024-09-16T02:14:40.098 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358741+0000", 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9580000000000001e-06 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358753+0000", 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1242e-05 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358833+0000", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9919999999999994e-05 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358861+0000", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8904999999999999e-05 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367924+0000", 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090629869999999998 2024-09-16T02:14:40.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367976+0000", 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1484999999999999e-05 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7011187999999997, 2024-09-16T02:14:40.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085149430000000005, 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359571+0000", 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.68e-06 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359582+0000", 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1399999999999999e-05 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359660+0000", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7704e-05 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359694+0000", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4062e-05 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368042+0000", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083476850000000005 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368084+0000", 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2412e-05 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7010376239999996, 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086774060000000004, 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.106 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359652+0000", 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.114e-06 2024-09-16T02:14:40.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359664+0000", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1617000000000001e-05 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359750+0000", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6180000000000005e-05 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359794+0000", 2024-09-16T02:14:40.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3962999999999999e-05 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368284+0000", 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084900930000000006 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368328+0000", 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3439000000000002e-05 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.7005664469999999, 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079952310000000002, 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360122+0000", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7699999999999992e-07 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360169+0000", 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6087000000000001e-05 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360275+0000", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106805 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360296+0000", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0744999999999999e-05 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360496+0000", 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020018400000000001 2024-09-16T02:14:40.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360529+0000", 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2329000000000002e-05 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368025+0000", 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074963929999999996 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368117+0000", 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1711000000000003e-05 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.69827818, 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079995459999999997, 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.114 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362411+0000", 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.53e-06 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362463+0000", 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1981999999999998e-05 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362640+0000", 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000177047 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362687+0000", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6349e-05 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365835+0000", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031487730000000001 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365859+0000", 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3612999999999999e-05 2024-09-16T02:14:40.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.370339+0000", 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044803380000000004 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.370409+0000", 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9913999999999995e-05 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.362629+0000", 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6980593270000002, 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0057981700000000001, 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362629+0000", 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362629+0000", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362630+0000", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8300000000000001e-06 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362642+0000", 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1353e-05 2024-09-16T02:14:40.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362792+0000", 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015000500000000001 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362816+0000", 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4363e-05 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368384+0000", 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055679830000000003 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368427+0000", 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2636000000000003e-05 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.364463+0000", 2024-09-16T02:14:40.122 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6962250690000005, 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040598220000000003, 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364463+0000", 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364463+0000", 2024-09-16T02:14:40.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364465+0000", 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364477+0000", 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1963e-05 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364561+0000", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3761e-05 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.364580+0000", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9355999999999999e-05 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368482+0000", 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039016929999999999 2024-09-16T02:14:40.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368523+0000", 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1048999999999999e-05 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.126 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.692903168, 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076441920000000002, 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:40.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.63e-07 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367837+0000", 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2117999999999998e-05 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372037+0000", 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042000819999999999 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372060+0000", 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2782999999999999e-05 2024-09-16T02:14:40.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372212+0000", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152206 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372234+0000", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1475999999999999e-05 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375366+0000", 2024-09-16T02:14:40.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031322160000000002 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375429+0000", 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2848000000000005e-05 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 45/32 ECSubWrite(tid=53, reqid=client.4598.0:250, at_version=45'45, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.377898+0000", 2024-09-16T02:14:40.131 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.682790131, 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0052514700000000003, 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.377898+0000", 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.377898+0000", 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.377900+0000", 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2000000000000001e-06 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.378057+0000", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015719799999999999 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.378152+0000", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4744e-05 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.378174+0000", 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1832999999999999e-05 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.378353+0000", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017896399999999999 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.378378+0000", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5295000000000001e-05 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.383061+0000", 2024-09-16T02:14:40.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046830450000000003 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.383149+0000", 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8190999999999996e-05 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=22, reqid=osd.10.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.380879+0000", 2024-09-16T02:14:40.136 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6798084949999996, 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046219770000000002, 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380879+0000", 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380879+0000", 2024-09-16T02:14:40.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380881+0000", 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.592e-06 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380905+0000", 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4349999999999999e-05 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380974+0000", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8392000000000005e-05 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380994+0000", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0299999999999999e-05 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381123+0000", 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012927699999999999 2024-09-16T02:14:40.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381139+0000", 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5404e-05 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.385430+0000", 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004291624 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.385501+0000", 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1038000000000007e-05 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=35, last_complete=45'32, committed=1, applied=1))", 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6789066679999998, 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65850650099999997, 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.141 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381783+0000", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.857e-06 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381795+0000", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1466e-05 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040081+0000", 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65828677099999999 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040095+0000", 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3686000000000001e-05 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040215+0000", 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011949 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040288+0000", 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3231000000000004e-05 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=24, reqid=client.4598.0:254, at_version=45'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6682081020000004, 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063935490000000001, 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.145 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.146 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392481+0000", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.282e-06 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392591+0000", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109607 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397038+0000", 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044475239999999996 2024-09-16T02:14:40.147 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397053+0000", 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4962e-05 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397183+0000", 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012977600000000001 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.148 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397207+0000", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3564e-05 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.398820+0000", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001613094 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.398873+0000", 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3739999999999997e-05 2024-09-16T02:14:40.149 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.663700575, 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27792197099999999, 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.150 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396988+0000", 2024-09-16T02:14:40.151 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759999999999999e-06 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397010+0000", 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1525000000000001e-05 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673630+0000", 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27662027 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.152 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673655+0000", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5003000000000001e-05 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673857+0000", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020175299999999999 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673875+0000", 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:14:40.153 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674832+0000", 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095631999999999998 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674909+0000", 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7741e-05 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.154 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.155 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.155 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.155 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:40.156 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:14:40.157 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:40.151+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:40.257 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:40.254+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:40.325 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:40.326 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:40.326 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:40.326 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:40.342 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:14:40.387 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:40.383+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:40.398 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:14:40.410 INFO:tasks.rados.rados.0.smithi195.stdout:152: done (0 left) 2024-09-16T02:14:40.410 INFO:tasks.rados.rados.0.smithi195.stdout:153: snap_create 2024-09-16T02:14:40.458 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:40.455+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:40.471 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:40.471 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:40.471 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:40.471 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:40.471 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:40.485 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:14:40.559 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:40.558+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:40.652 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:40.652 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:40.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:40.652 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:40.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.318459+0000", 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.330884673, 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090249379999999997, 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.318459+0000", 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.318459+0000", 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.318459+0000", 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.318984+0000", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052479000000000002 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.319038+0000", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3594999999999998e-05 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.319056+0000", 2024-09-16T02:14:40.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8566000000000001e-05 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.319230+0000", 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017405599999999999 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.319259+0000", 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9145999999999998e-05 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327426+0000", 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081660440000000008 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327484+0000", 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8121000000000001e-05 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=17, reqid=client.4598.0:193, at_version=38'11, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:40.657 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.323708+0000", 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.325634889, 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038565639999999998, 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.323708+0000", 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.323708+0000", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.323711+0000", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.227e-06 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.323968+0000", 2024-09-16T02:14:40.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025695999999999998 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.324060+0000", 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1970000000000002e-05 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.324082+0000", 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2391999999999998e-05 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.324230+0000", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000148214 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.324254+0000", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4079000000000001e-05 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327493+0000", 2024-09-16T02:14:40.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032388479999999999 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327565+0000", 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1873999999999997e-05 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 38/32 ECSubWrite(tid=18, reqid=client.4598.0:194, at_version=38'12, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.326592+0000", 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.322750958, 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050027769999999999, 2024-09-16T02:14:40.662 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326592+0000", 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326592+0000", 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326594+0000", 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.714e-06 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326636+0000", 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1980999999999997e-05 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326732+0000", 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5394000000000002e-05 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.326754+0000", 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2858999999999999e-05 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327637+0000", 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088255899999999999 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.327651+0000", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3946e-05 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331489+0000", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038381449999999998 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331595+0000", 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106179 2024-09-16T02:14:40.666 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 38/32 ECSubWrite(tid=13, reqid=client.4598.0:196, at_version=38'8, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.329435+0000", 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.319908384, 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041558860000000001, 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329435+0000", 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329435+0000", 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329436+0000", 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7999999999999995e-07 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329482+0000", 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6573999999999998e-05 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329695+0000", 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021309599999999999 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329719+0000", 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4082000000000001e-05 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329877+0000", 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000157954 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329902+0000", 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4165999999999999e-05 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.333493+0000", 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035915610000000001 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.333591+0000", 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7773000000000001e-05 2024-09-16T02:14:40.671 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 38/32 ECSubWrite(tid=14, reqid=client.4598.0:197, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.329498+0000", 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.319845775999999, 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048716109999999996, 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329498+0000", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329498+0000", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329498+0000", 2024-09-16T02:14:40.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9299999999999997e-07 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329529+0000", 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0868000000000003e-05 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329791+0000", 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026174 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329810+0000", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9239999999999999e-05 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331166+0000", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013561599999999999 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.331195+0000", 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8469000000000001e-05 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.334218+0000", 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030228030000000001 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.334369+0000", 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000151638 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 38/32 ECSubWrite(tid=15, reqid=client.4598.0:198, at_version=38'10, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.329534+0000", 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.319809205, 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048968639999999999, 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329534+0000", 2024-09-16T02:14:40.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329534+0000", 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329535+0000", 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0299999999999999e-07 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329572+0000", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7066000000000001e-05 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329852+0000", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028046899999999999 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.329871+0000", 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9035999999999999e-05 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.332578+0000", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027071949999999999 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.332603+0000", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4779000000000001e-05 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.334381+0000", 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017778850000000001 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.334431+0000", 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9931e-05 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs4 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.335083+0000", 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.314260558, 2024-09-16T02:14:40.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030860919999999999, 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335083+0000", 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335083+0000", 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335083+0000", 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0199999999999998e-07 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335112+0000", 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8833e-05 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335144+0000", 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1642e-05 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335153+0000", 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2539999999999998e-06 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335264+0000", 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000111426 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.335283+0000", 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8961999999999999e-05 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.338134+0000", 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028507440000000001 2024-09-16T02:14:40.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.338169+0000", 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4928999999999997e-05 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 45/32 ECSubWrite(tid=31, reqid=osd.5.0:31, at_version=45'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.341646+0000", 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3076971510000002, 2024-09-16T02:14:40.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033606629999999998, 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341646+0000", 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341646+0000", 2024-09-16T02:14:40.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341647+0000", 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341702+0000", 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4792e-05 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341845+0000", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014304500000000001 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.341868+0000", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3278999999999999e-05 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.342106+0000", 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000238323 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.342127+0000", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0511999999999999e-05 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344915+0000", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0027884920000000001 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345007+0000", 2024-09-16T02:14:40.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1566999999999999e-05 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 45/32 ECSubWrite(tid=21, reqid=osd.6.0:21, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.343965+0000", 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3053780049999997, 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047659119999999998, 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.343965+0000", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.343965+0000", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.343966+0000", 2024-09-16T02:14:40.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1299999999999998e-07 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344042+0000", 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5964000000000007e-05 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344168+0000", 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012552 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344210+0000", 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1825e-05 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344334+0000", 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012478700000000001 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.344352+0000", 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8037999999999999e-05 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.348659+0000", 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043063470000000003 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.348731+0000", 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2517999999999999e-05 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.346495+0000", 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3028487520000001, 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032768179999999999, 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346495+0000", 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346495+0000", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346496+0000", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6500000000000001e-06 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346526+0000", 2024-09-16T02:14:40.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9975e-05 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346617+0000", 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0427999999999995e-05 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346641+0000", 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4145999999999999e-05 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346775+0000", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000134225 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.346797+0000", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1506e-05 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349633+0000", 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028364839999999998 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349771+0000", 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013840400000000001 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.349906+0000", 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2994377740000003, 2024-09-16T02:14:40.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035458379999999999, 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349906+0000", 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349906+0000", 2024-09-16T02:14:40.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349906+0000", 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9299999999999998e-07 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.349952+0000", 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5416e-05 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.350074+0000", 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012243599999999999 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.350096+0000", 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2011999999999999e-05 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.350235+0000", 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013920999999999999 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.350260+0000", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4552000000000001e-05 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.353349+0000", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030892889999999998 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.353451+0000", 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010233 2024-09-16T02:14:40.704 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 45/32 ECSubWrite(tid=22, reqid=osd.6.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351041+0000", 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.298302047, 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025094879999999998, 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351041+0000", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351041+0000", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351043+0000", 2024-09-16T02:14:40.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.734e-06 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351067+0000", 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3572000000000001e-05 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351142+0000", 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5680999999999996e-05 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351164+0000", 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1539000000000002e-05 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351303+0000", 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013919599999999999 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351325+0000", 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1851000000000001e-05 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.708 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.353481+0000", 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002155807 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.353551+0000", 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0108e-05 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.709 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.355889+0000", 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2934546200000003, 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036037500000000002, 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.355889+0000", 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.355889+0000", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.355889+0000", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6700000000000003e-07 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.355938+0000", 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9020999999999997e-05 2024-09-16T02:14:40.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356064+0000", 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012568599999999999 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356085+0000", 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1223e-05 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356225+0000", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000139534 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356248+0000", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3082e-05 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359382+0000", 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031338120000000001 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359493+0000", 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110825 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs1 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357808+0000", 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2915351470000003, 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029741250000000002, 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.714 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357808+0000", 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357808+0000", 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357809+0000", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8000000000000005e-07 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357856+0000", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7191000000000003e-05 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357964+0000", 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010814700000000001 2024-09-16T02:14:40.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357988+0000", 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4068999999999999e-05 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358340+0000", 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035158900000000002 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358373+0000", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2945999999999999e-05 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360689+0000", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0023158829999999999 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360782+0000", 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3519999999999999e-05 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.718 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/32 ECSubWrite(tid=34, reqid=osd.3.0:34, at_version=45'32, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.361861+0000", 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2874825830000001, 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044698100000000003, 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.361861+0000", 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.361861+0000", 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.361862+0000", 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5519999999999999e-06 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.361910+0000", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7258000000000001e-05 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.361993+0000", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2978000000000004e-05 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362013+0000", 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:14:40.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039762199999999998 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362450+0000", 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9456000000000003e-05 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366243+0000", 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037927159999999998 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366331+0000", 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8052000000000003e-05 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s5 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.368379+0000", 2024-09-16T02:14:40.723 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2809640350000002, 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004755815, 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368379+0000", 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368379+0000", 2024-09-16T02:14:40.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368380+0000", 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9100000000000004e-07 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368425+0000", 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5235999999999999e-05 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368528+0000", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103111 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368548+0000", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0102000000000002e-05 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368695+0000", 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014657 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368718+0000", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2799e-05 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.373047+0000", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043288060000000001 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.373135+0000", 2024-09-16T02:14:40.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8599999999999999e-05 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs1 45/32 ECSubWrite(tid=9, reqid=osd.14.0:8, at_version=0'0, trim_to=0'0, roll_forward_to=45'8))", 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.368480+0000", 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2808638600000002, 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00463417, 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.728 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368480+0000", 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368480+0000", 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368481+0000", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368516+0000", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4649e-05 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368599+0000", 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2880999999999995e-05 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368623+0000", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3788999999999999e-05 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368785+0000", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000161982 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368808+0000", 2024-09-16T02:14:40.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3606000000000001e-05 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.373025+0000", 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042170059999999997 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.373114+0000", 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8367000000000003e-05 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.732 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs4 45/32 ECSubWrite(tid=25, reqid=osd.2.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.374462+0000", 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2748812689999998, 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022197760000000001, 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374462+0000", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374462+0000", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374464+0000", 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.99e-06 2024-09-16T02:14:40.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374493+0000", 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9379000000000001e-05 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374608+0000", 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000114676 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374634+0000", 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5795999999999998e-05 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374815+0000", 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000181304 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374840+0000", 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4493000000000001e-05 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.376595+0000", 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0017550560000000001 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.376682+0000", 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7082000000000004e-05 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s3 45/32 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=45'32))", 2024-09-16T02:14:40.737 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.374708+0000", 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2746349869999998, 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022862569999999999, 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374708+0000", 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374708+0000", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374710+0000", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.888e-06 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374737+0000", 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6414000000000002e-05 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374834+0000", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7483000000000002e-05 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.374856+0000", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1722e-05 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375055+0000", 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019885200000000001 2024-09-16T02:14:40.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375078+0000", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3546000000000001e-05 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.376932+0000", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018540480000000001 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.376995+0000", 2024-09-16T02:14:40.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2304000000000005e-05 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s5 45/32 ECSubWrite(tid=23, reqid=client.4598.0:253, at_version=45'14, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.387996+0000", 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.2613472860000003, 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025231440000000002, 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:40.742 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.387996+0000", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.387996+0000", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.387998+0000", 2024-09-16T02:14:40.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4580000000000001e-06 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.388877+0000", 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00087918200000000005 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.388960+0000", 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2888999999999999e-05 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.388985+0000", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4998999999999999e-05 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.389069+0000", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.389078+0000", 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1430000000000004e-06 2024-09-16T02:14:40.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.390441+0000", 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001363036 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.390519+0000", 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8207000000000005e-05 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:40.746 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:40.748 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:14:40.749 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:40.747+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:40.849 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:40.846+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:153: done (0 left) 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:154: rollback oid 26 current snap is 10 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 26 to 6 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:155: copy_from oid 18 from oid 15 current snap is 10 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 8 current snap is 10 2024-09-16T02:14:40.906 INFO:tasks.rados.rados.0.smithi195.stdout:156: seq_num 80 ranges {1163264=589824,1753088=573440,2326528=196608} 2024-09-16T02:14:40.923 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:40.923 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:40.923 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:40.923 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:40.937 INFO:tasks.rados.rados.0.smithi195.stdout:156: writing smithi19530025-8 from 1163264 to 1753088 tid 1 2024-09-16T02:14:40.939 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:14:40.947 INFO:tasks.rados.rados.0.smithi195.stdout:156: writing smithi19530025-8 from 1753088 to 2326528 tid 2 2024-09-16T02:14:40.950 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:40.950+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:40.951 INFO:tasks.rados.rados.0.smithi195.stdout:156: writing smithi19530025-8 from 2326528 to 2523136 tid 3 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:154: finishing rollback tid 0 to smithi19530025-26 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:154: finishing rollback tid 1 to smithi19530025-26 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:154: finishing rollback tid 2 to smithi19530025-26 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 26 v 25 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:155: finishing copy_from racing read to smithi19530025-18 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:156: finishing write tid 1 to smithi19530025-8 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:154: done (2 left) 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 47 current snap is 10 2024-09-16T02:14:40.952 INFO:tasks.rados.rados.0.smithi195.stdout:157: seq_num 81 ranges {0=606208,606208=720896,1327104=49152} 2024-09-16T02:14:40.961 INFO:tasks.rados.rados.0.smithi195.stdout:157: writing smithi19530025-47 from 0 to 606208 tid 1 2024-09-16T02:14:40.974 INFO:tasks.rados.rados.0.smithi195.stdout:157: writing smithi19530025-47 from 606208 to 1327104 tid 2 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:157: writing smithi19530025-47 from 1327104 to 1376256 tid 3 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:158: read oid 28 snap 8 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:158: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 19 current snap is 10 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:156: finishing write tid 2 to smithi19530025-8 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:156: finishing write tid 3 to smithi19530025-8 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:156: finishing write tid 4 to smithi19530025-8 2024-09-16T02:14:40.975 INFO:tasks.rados.rados.0.smithi195.stdout:159: seq_num 82 ranges {311296=622592,933888=475136,1409024=294912} 2024-09-16T02:14:40.989 INFO:tasks.rados.rados.0.smithi195.stdout:159: writing smithi19530025-19 from 311296 to 933888 tid 1 2024-09-16T02:14:40.995 INFO:tasks.rados.rados.0.smithi195.stdout:159: writing smithi19530025-19 from 933888 to 1409024 tid 2 2024-09-16T02:14:41.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: writing smithi19530025-19 from 1409024 to 1703936 tid 3 2024-09-16T02:14:41.000 INFO:tasks.rados.rados.0.smithi195.stdout:160: read oid 4 snap -1 2024-09-16T02:14:41.000 INFO:tasks.rados.rados.0.smithi195.stdout:160: expect deleted 2024-09-16T02:14:41.000 INFO:tasks.rados.rados.0.smithi195.stdout:161: snap_remove snap 9 2024-09-16T02:14:41.051 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:41.043+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:41.064 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:41.064 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:41.065 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:41.065 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:41.065 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:41.079 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:14:41.167 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:41.163+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:41.249 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:41.249 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:31 3.1s0 3:89ce0a26:::smithi19530025-11:head [append 0~180224,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.955408050999999, 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022356674999999999, 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.250 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 31 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3159999999999999e-06 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290984+0000", 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032549899999999998 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290996+0000", 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2635e-05 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291004+0000", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5440000000000001e-06 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291070+0000", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6177e-05 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291178+0000", 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107766 2024-09-16T02:14:41.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295206+0000", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040278029999999999 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299290+0000", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040844139999999998 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312901+0000", 2024-09-16T02:14:41.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013611165999999999 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313013+0000", 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011135499999999999 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:32 3.1s0 3:89ce0a26:::smithi19530025-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:41.255 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.955032719999998, 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022289759999999999, 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 32 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:41.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291032+0000", 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2600000000000001e-07 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291036+0000", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2760000000000002e-06 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291044+0000", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7190000000000001e-06 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291050+0000", 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5779999999999998e-06 2024-09-16T02:14:41.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295781+0000", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047316909999999997 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295883+0000", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010134 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296411+0000", 2024-09-16T02:14:41.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052795499999999998 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299292+0000", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028809719999999999 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313180+0000", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013887932 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313321+0000", 2024-09-16T02:14:41.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141371 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:33 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.955000496, 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024514080000000001, 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.261 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 33 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291064+0000", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291067+0000", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7199999999999998e-06 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291074+0000", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4680000000000003e-06 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291080+0000", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.964e-06 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055391950000000002 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0838000000000001e-05 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313405+0000", 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016704404999999999 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313545+0000", 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014004999999999999 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315578+0000", 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020328199999999999 2024-09-16T02:14:41.265 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:37 3.1s0 3:81c6c0a8:::smithi19530025-13:head [append 0~212992,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954972048999998, 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023064575, 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.266 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 37 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291093+0000", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4399999999999999e-07 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291317+0000", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022428500000000001 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291335+0000", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7980999999999999e-05 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291347+0000", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1708e-05 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296736+0000", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053889660000000002 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296859+0000", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123786 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297520+0000", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066093900000000004 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302005+0000", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044842199999999997 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314113+0000", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012108659000000001 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314156+0000", 2024-09-16T02:14:41.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3186999999999998e-05 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:38 3.1s0 3:81c6c0a8:::smithi19530025-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954702564000002, 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023044124999999999, 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.271 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 38 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291362+0000", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0100000000000001e-06 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291368+0000", 2024-09-16T02:14:41.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6879999999999996e-06 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291381+0000", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3369e-05 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291404+0000", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2164e-05 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298518+0000", 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007114029 2024-09-16T02:14:41.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298598+0000", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0813e-05 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299017+0000", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041827500000000002 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302007+0000", 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029904630000000001 2024-09-16T02:14:41.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314313+0000", 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012305768999999999 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314405+0000", 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2545000000000002e-05 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:39 3.1s0 3:81c6c0a8:::smithi19530025-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.276 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954630631000001, 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024374809000000001, 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 39 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291434+0000", 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:14:41.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291440+0000", 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4960000000000004e-06 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291456+0000", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025e-05 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291468+0000", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299421+0000", 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079526309999999999 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299510+0000", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9506000000000007e-05 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314495+0000", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014984548 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314572+0000", 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6854999999999995e-05 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315808+0000", 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012362639999999999 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:40 3.1s0 3:830a71a8:::smithi19530025-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954579038999999, 2024-09-16T02:14:41.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024497693000000001, 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 40 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291486+0000", 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8999999999999995e-07 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291773+0000", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028741999999999998 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291795+0000", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.181e-05 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291807+0000", 2024-09-16T02:14:41.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1687e-05 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299547+0000", 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077406699999999998 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299647+0000", 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9978999999999999e-05 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.300445+0000", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079766400000000001 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302061+0000", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016158539999999999 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315913+0000", 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013852573999999999 2024-09-16T02:14:41.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315983+0000", 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9145000000000003e-05 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:41 3.1s0 3:830a71a8:::smithi19530025-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954231705000002, 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024477925000000001, 2024-09-16T02:14:41.288 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291833+0000", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291839+0000", 2024-09-16T02:14:41.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291855+0000", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291865+0000", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0037999999999999e-05 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301194+0000", 2024-09-16T02:14:41.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093284930000000002 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301281+0000", 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6972999999999996e-05 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301794+0000", 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051343099999999996 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.304892+0000", 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030976950000000001 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316223+0000", 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011331071999999999 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316310+0000", 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.742e-05 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.293 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:42 3.1s0 3:830a71a8:::smithi19530025-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.954185512999999, 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031481841000000003, 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.294 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291880+0000", 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2139999999999999e-06 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291883+0000", 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291902+0000", 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8881000000000001e-05 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291912+0000", 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9680000000000001e-06 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302284+0000", 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371521999999999 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302334+0000", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0047999999999999e-05 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318985+0000", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016650888999999999 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.319121+0000", 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136336 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323360+0000", 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042391670000000003 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:64 3.1s0 3:8151366d:::smithi19530025-22:head [append 0~360448,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.927720911000002, 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014450049, 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.299 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 64 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318343+0000", 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318345+0000", 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.75e-06 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318434+0000", 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9405000000000002e-05 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318448+0000", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4042999999999999e-05 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318456+0000", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9289999999999998e-06 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.320462+0000", 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002006271 2024-09-16T02:14:41.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.320696+0000", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000233231 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.321663+0000", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096740900000000002 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323488+0000", 2024-09-16T02:14:41.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001825089 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332528+0000", 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090399920000000002 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332793+0000", 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026493000000000003 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:65 3.1s0 3:8151366d:::smithi19530025-22:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.927589817000001, 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014507262999999999, 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 65 2024-09-16T02:14:41.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318474+0000", 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318475+0000", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3600000000000003e-07 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318480+0000", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3000000000000001e-06 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318488+0000", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4479999999999997e-06 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318493+0000", 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7200000000000003e-06 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322383+0000", 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0038899780000000001 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322473+0000", 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0278000000000005e-05 2024-09-16T02:14:41.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.322933+0000", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00045975500000000001 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323635+0000", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00070165699999999995 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332907+0000", 2024-09-16T02:14:41.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092716980000000001 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.332981+0000", 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4793e-05 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:66 3.1s0 3:8151366d:::smithi19530025-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.927560893999999, 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016162184999999999, 2024-09-16T02:14:41.310 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 66 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318503+0000", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318504+0000", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3799999999999997e-07 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318507+0000", 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7469999999999999e-06 2024-09-16T02:14:41.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318515+0000", 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2250000000000003e-06 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318521+0000", 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3269999999999997e-06 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323407+0000", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004885777 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323480+0000", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3491000000000005e-05 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.333374+0000", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098940399999999998 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.333475+0000", 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010128099999999999 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.334665+0000", 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011897590000000001 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:244 3.1s0 3:89ce0a26:::smithi19530025-11:head [create,setxattr STaaXsn_tt`[noPeao^kO]r]qhc]OnRdXWqXa]RQ (421) in=40b,setxattr QccpemYdNR_fUkaT_Zk (1613) in=19b,setxattr e]rWtr\\spa\\[`\\etXP]UVYl (1613) in=23b,setxattr ctjQdgZSs[oQ^YekRX]`P_kg`lYfeo\\]jOW\\Ncgj (105) in=40b,setxattr ]UUsac^QNTgnq\\c_amO]eQ^UmNmdPVNbpsche]nq (1366) in=40b,setxattr tsV]fVpQmWmtatroVb]qjW\\n\\^hTba^e (1613) in=32b,setxattr plUORnNkpnPqade]gn^TNqootWi^\\Ygrc[O[dZfX (105) in=40b,setxattr ]aq]PelqnUhOnhUcccRY_o`UqoXf`_^fRi`adYdm (630) in=40b,setxattr lobeSZbbYgcdsPag__QR]`_ih[ (1613) in=26b,setxattr cmbU]Vpc[onnejbfSlj]qNT]`Yspgre^bQ[go^so (138) in=40b,setxattr pkmgTn^hlhmYnjkbSslNYhkOctl_ctnq]^Wm (1613) in=36b,setxattr jrdhlkfStVpRdk]N[rjbfWgQ_RYkY^ZYdg]\\l[dV (1367) in=40b,setxattr `jS`f[NpneY_j`inTVdpWq`ntO_diaj[folX`ibo (1473) in=40b,setxattr aj_mddZOpYlRadmQjbSnSRVdY\\gTgZ]RNfWSSbbo (1858) in=40b,setxattr YegfaljqWjdPp\\]^ejiXpWObnZd]dQTlihpU_Q`q (1649) in=40b,setxattr UsbSqlQaZndaoThjTR\\i_ei]Zs[hUpZasNRfZkaO (1611) in=40b,setxattr oZqjdnYqTXfkcaTmagUaNpZTgXfVkphqmiUTlmOf (1612) in=40b,setxattr [PdlRk]`\\iiigl[fRo`obkeWlZ]W`eqS[nmRn[kg (841) in=40b,setxattr eQXfldiUZkeo^Sjbkb_shi[kNcVQe^W[loVl[g^d (1753) in=40b,setxattr aYNbp]cNSQRqR\\PWsdUPp[Qp]`Zl\\fdnSPeUOdn\\ (13964) in=40b] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.417323+0000", 2024-09-16T02:14:41.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.8287408459999996, 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017331537000000001, 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 244 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417323+0000", 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417323+0000", 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417325+0000", 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3180000000000001e-06 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417353+0000", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7552e-05 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417376+0000", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3368999999999999e-05 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417392+0000", 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5305e-05 2024-09-16T02:14:41.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417502+0000", 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109907 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.417673+0000", 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000171273 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.418445+0000", 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00077169300000000005 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.419716+0000", 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012717270000000001 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434562+0000", 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014845763 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434655+0000", 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2629999999999999e-05 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:247 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (56) in=7b] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.8155027009999998, 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30233062700000002, 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.321 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 247 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430562+0000", 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1230000000000001e-06 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430568+0000", 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.519e-06 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430582+0000", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.366e-05 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430592+0000", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0482999999999999e-05 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433637+0000", 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003044705 2024-09-16T02:14:41.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433734+0000", 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6916000000000005e-05 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434127+0000", 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039313800000000002 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435690+0000", 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015630640000000001 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732747+0000", 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29705759700000001 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732892+0000", 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014442200000000001 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:248 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.8154282579999999, 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.304991454, 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 248 2024-09-16T02:14:41.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430637+0000", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3099999999999996e-07 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430641+0000", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430654+0000", 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638999999999999e-05 2024-09-16T02:14:41.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430662+0000", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7370000000000006e-06 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434972+0000", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043098019999999997 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435055+0000", 2024-09-16T02:14:41.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3625e-05 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.733794+0000", 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.298738691 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.734027+0000", 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023328199999999999 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.735627+0000", 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015998379999999999 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:250 3.1s0 3:89ce0a26:::smithi19530025-11:head [copy-from ver 2] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.9010035060000003, 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046806635999999999, 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.332 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 250 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:41.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345064+0000", 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.484e-06 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345079+0000", 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4939999999999999e-05 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345114+0000", 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5258e-05 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345131+0000", 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7356000000000001e-05 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345312+0000", 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018022200000000001 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345332+0000", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0509e-05 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011526471 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357011+0000", 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015221500000000001 2024-09-16T02:14:41.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375994+0000", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018982974999999999 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380796+0000", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048023010000000001 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391742+0000", 2024-09-16T02:14:41.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010946141 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391867+0000", 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124764 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:251 3.1s0 3:89ce0a26:::smithi19530025-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:41.338 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.9008859000000005, 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034174018, 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 251 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345179+0000", 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1799999999999999e-06 2024-09-16T02:14:41.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345186+0000", 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5259999999999999e-06 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345202+0000", 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6453999999999999e-05 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345213+0000", 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1147e-05 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345556+0000", 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000342219 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345581+0000", 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5876000000000001e-05 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012465654 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358190+0000", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014254599999999999 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379056+0000", 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020865887 2024-09-16T02:14:41.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379212+0000", 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156675 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379352+0000", 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013985399999999999 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 45/32 ECSubWriteReply(tid=55, last_complete=45'49, committed=1, applied=1))", 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.8530668840000004, 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.456257309, 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392998+0000", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2780000000000001e-06 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.393006+0000", 2024-09-16T02:14:41.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1310000000000006e-06 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847561+0000", 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45455489999999998 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847576+0000", 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.453e-05 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847695+0000", 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118799 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.849254+0000", 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015596710000000001 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:260 3.1s0 3:830a71a8:::smithi19530025-14:head [copy-from ver 2] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.8889553179999998, 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02650228, 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 260 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357110+0000", 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.243e-06 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357117+0000", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9e-06 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357132+0000", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5156e-05 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357141+0000", 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-16T02:14:41.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357434+0000", 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029301799999999998 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357457+0000", 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2821999999999999e-05 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365347+0000", 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007890088 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365650+0000", 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030261099999999999 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.379802+0000", 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014151762 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381743+0000", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001941531 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383415+0000", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016721349999999999 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383611+0000", 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019561399999999999 2024-09-16T02:14:41.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:261 3.1s0 3:830a71a8:::smithi19530025-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.8888744910000002, 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024496032000000001, 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 261 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357191+0000", 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3799999999999999e-06 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357198+0000", 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1740000000000003e-06 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357220+0000", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2322000000000001e-05 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357238+0000", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7377e-05 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357578+0000", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034031900000000001 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:41.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357600+0000", 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1543e-05 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366106+0000", 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085068360000000003 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366241+0000", 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013419899999999999 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381121+0000", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014880769 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381357+0000", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023564300000000001 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381685+0000", 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032846999999999998 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.360 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.361 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.361 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:41.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:41.362 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:41.358+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:41.363 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:14:41.378 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:14:41.378 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:14:41.378 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-16T02:14:41.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-16T02:14:41.463 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:41.459+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:41.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.62841381100000004, 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62846478900000002, 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:41.535 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:41.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:14:41.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:14:41.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: ], 2024-09-16T02:14:41.540 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 1 2024-09-16T02:14:41.541 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:41.549 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:14:41.556 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:41.556 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:41.564 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:41.562+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:41.582 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:41.582 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:41.620 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:41.621 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:41.657 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:41.664 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:41.659+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:41.697 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:41.697 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:41.714 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:41.714 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:41.714 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:41.714 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:41.714 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:41.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:41.723 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:41.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:14:41.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:41.748 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 52, 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "babbcdda-abe3-4b2f-82b4-da47bc408d34", 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-09-16T02:13:51.176653+0000", 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-09-16T02:14:40.895847+0000", 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-09-16T02:14:04.035127+0000", 2024-09-16T02:14:41.763 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-09-16T02:13:55.736760+0000", 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 6, 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T02:14:41.764 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 3, 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 16, 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "jewel", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-09-16T02:14:04.065882+0000", 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-09-16T02:14:41.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "is_stretch_pool": false, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2024-09-16T02:14:41.766 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "25", 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-09-16T02:14:41.767 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-09-16T02:14:41.768 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-09-16T02:14:41.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.770 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 15.380000114440918, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 15.380000114440918, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.12999999523162842, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "rbd", 2024-09-16T02:14:41.771 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-09-16T02:14:06.221379+0000", 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 2, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 1, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "is_stretch_pool": false, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-09-16T02:14:41.772 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 8, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 8, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 8, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 8, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 8, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-09-16T02:14:41.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "28", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 28, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-09-16T02:14:41.774 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-09-16T02:14:41.775 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-09-16T02:14:41.776 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 4.0199999809265137, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 4.0199999809265137, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.56000000238418579, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_0", 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-09-16T02:14:20.234400+0000", 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 3, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 7, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 5, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 1, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "is_stretch_pool": false, 2024-09-16T02:14:41.778 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 16, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 16, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 16, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 16, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 16, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-09-16T02:14:41.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "52", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 11, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 52, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-09-16T02:14:41.780 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "shecprofile", 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-09-16T02:14:41.781 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 16384, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.782 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "d00dcd0d-0b5f-48c9-8d98-adaaabf5283a", 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.783 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.784 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.785 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3190094522 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3190094522", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6811/3190094522", 2024-09-16T02:14:41.787 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6815/3190094522", 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6813/3190094522", 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c", 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.788 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 0, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6816", 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6817", 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6818", 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6819", 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6822", 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6823", 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6820", 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6821", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1191817370 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6817/1191817370", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6819/1191817370", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6823/1191817370", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6821/1191817370", 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.793 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "6bfe62d5-9c5b-44df-bc4f-cf312837c932", 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.794 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6816", 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6817", 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6818", 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6819", 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6822", 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.797 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6823", 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6820", 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6821", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3704126567 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.177:6817/3704126567", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.177:6819/3704126567", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.177:6823/3704126567", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.177:6821/3704126567", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.799 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "fe2b6f49-91ab-4692-ab9c-4b3bda3bf429", 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.800 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6826", 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6827", 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6828", 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.802 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6829", 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6832", 2024-09-16T02:14:41.803 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6833", 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.804 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6830", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6831", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3376854549 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.195:6827/3376854549", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.195:6829/3376854549", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.195:6833/3376854549", 2024-09-16T02:14:41.805 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.195:6831/3376854549", 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "399375fc-ce28-40e2-89b1-e0fc2c1f44ec", 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.806 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 0, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6824", 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6825", 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6826", 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6827", 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6830", 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6831", 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6828", 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6829", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1858329864 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6825/1858329864", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6827/1858329864", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6831/1858329864", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6829/1858329864", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.811 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3", 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.812 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.815 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2650650539 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6801/2650650539", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6803/2650650539", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/2650650539", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6805/2650650539", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ac6efa66-27ed-494c-956d-22b90be7a8fd", 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.818 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6824", 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6825", 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.819 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6826", 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6827", 2024-09-16T02:14:41.820 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6830", 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.821 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6831", 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6828", 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.822 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6829", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 47176522 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.177:6825/47176522", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.177:6827/47176522", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.177:6831/47176522", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.177:6829/47176522", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "fb2a81f7-b4f5-4c4c-aff8-801e8b73d192", 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.824 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6810", 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6811", 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6812", 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6813", 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.826 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6816", 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6817", 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.827 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6814", 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6815", 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3756385243 2024-09-16T02:14:41.828 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.195:6811/3756385243", 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.195:6813/3756385243", 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.195:6817/3756385243", 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.195:6815/3756385243", 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.829 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 8, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "01aca5fb-932f-4ecb-a9ab-c716a5247373", 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.830 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6817", 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.831 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6818", 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6819", 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.832 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6822", 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6823", 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6820", 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6821", 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3976393740 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.834 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6817/3976393740", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6819/3976393740", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6823/3976393740", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6821/3976393740", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 9, 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "d52efa47-5645-4018-b495-ed116da5f7a9", 2024-09-16T02:14:41.835 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 0, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.836 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6824", 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6825", 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6826", 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6827", 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6830", 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6831", 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.839 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6828", 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6829", 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2321182247 2024-09-16T02:14:41.840 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6825/2321182247", 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6827/2321182247", 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6831/2321182247", 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6829/2321182247", 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.841 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 10, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "99005d53-e62d-4c20-8d3c-541fd20cbeb8", 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6808", 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6809", 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.843 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6810", 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6811", 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.844 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6814", 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6815", 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6812", 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6813", 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3769110015 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.846 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.177:6809/3769110015", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.177:6811/3769110015", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.177:6815/3769110015", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.177:6813/3769110015", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 11, 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "f460f1c7-0172-4bd9-875f-b34aff85804f", 2024-09-16T02:14:41.847 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.848 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6818", 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6819", 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6820", 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6821", 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6824", 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6825", 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6822", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6823", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1553304829 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.195:6819/1553304829", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.195:6821/1553304829", 2024-09-16T02:14:41.852 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.195:6825/1553304829", 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.195:6823/1553304829", 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 12, 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "7bd5fea1-6e94-478a-8a8f-87459b4030a2", 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.853 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6800", 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-09-16T02:14:41.857 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3221525577 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/3221525577", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6803/3221525577", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6807/3221525577", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6805/3221525577", 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.858 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 13, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "89c43816-fffc-42a6-98af-7f215631d595", 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 22, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.859 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-09-16T02:14:41.862 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-09-16T02:14:41.863 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 84283894 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6809/84283894", 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6811/84283894", 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/84283894", 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6813/84283894", 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 14, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "33cb3383-2d68-4928-be43-cfc215f6b080", 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 21, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.865 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6800", 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6801", 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6802", 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6803", 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6806", 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6807", 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6804", 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.177:6805", 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1752008388 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.869 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.177:6801/1752008388", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.177:6803/1752008388", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.177:6807/1752008388", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.177:6805/1752008388", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 15, 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e", 2024-09-16T02:14:41.870 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 21, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 32, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6802", 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6803", 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6804", 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6805", 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-09-16T02:14:41.873 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6808", 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6809", 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.874 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6806", 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.195:6807", 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3564981919 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.195:6803/3564981919", 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.195:6805/3564981919", 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.195:6809/3564981919", 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.195:6807/3564981919", 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.876 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.390713+0000", 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.877 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.329190+0000", 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.878 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.172914+0000", 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.553892+0000", 2024-09-16T02:14:41.879 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.795237+0000", 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.880 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:01.952515+0000", 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.881 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.696414+0000", 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.100374+0000", 2024-09-16T02:14:41.882 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 8, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.720464+0000", 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 9, 2024-09-16T02:14:41.883 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.793071+0000", 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 10, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.884 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:01.982210+0000", 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 11, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.697339+0000", 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 12, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.478801+0000", 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 13, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:02.343858+0000", 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 14, 2024-09-16T02:14:41.887 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:01.923086+0000", 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 15, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540719139924082687, 2024-09-16T02:14:41.888 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-09-16T02:14:01.888880+0000", 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": {}, 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-09-16T02:14:41.889 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "shecprofile": { 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "c": "2", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-device-class": "", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-num-failure-domains": "0", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-osds-per-failure-domain": "0", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-root": "default", 2024-09-16T02:14:41.890 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "4", 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "3", 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "shecprofile", 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "shec", 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "multiple" 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [ 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "snaps": [ 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "begin": 7, 2024-09-16T02:14:41.891 INFO:teuthology.orchestra.run.smithi003.stdout: "length": 1 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "begin": 9, 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "length": 1 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [ 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: "snaps": [ 2024-09-16T02:14:41.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "begin": 2, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "length": 5 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-09-16T02:14:41.893 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-09-16T02:14:41.894 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-09-16T02:14:41.894 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:41.894 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:41.895 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:41.895 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:41.897 INFO:tasks.thrashosds.thrasher:j is {'epoch': 52, 'fsid': 'babbcdda-abe3-4b2f-82b4-da47bc408d34', 'created': '2024-09-16T02:13:51.176653+0000', 'modified': '2024-09-16T02:14:40.895847+0000', 'last_up_change': '2024-09-16T02:14:04.035127+0000', 'last_in_change': '2024-09-16T02:13:55.736760+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 16, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-16T02:14:04.065882+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 15.380000114440918, 'score_stable': 15.380000114440918, 'optimal_score': 0.12999999523162842, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-16T02:14:06.221379+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '28', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 28, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.019999980926514, 'score_stable': 4.019999980926514, 'optimal_score': 0.5600000023841858, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-16T02:14:20.234400+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 3, 'size': 7, 'min_size': 5, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 11, 'snap_epoch': 52, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'shecprofile', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 16384, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'd00dcd0d-0b5f-48c9-8d98-adaaabf5283a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 3190094522}, {'type': 'v1', 'addr': '172.21.15.3:6809', 'nonce': 3190094522}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3190094522}, {'type': 'v1', 'addr': '172.21.15.3:6811', 'nonce': 3190094522}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 3190094522}, {'type': 'v1', 'addr': '172.21.15.3:6815', 'nonce': 3190094522}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3190094522}, {'type': 'v1', 'addr': '172.21.15.3:6813', 'nonce': 3190094522}]}, 'public_addr': '172.21.15.3:6809/3190094522', 'cluster_addr': '172.21.15.3:6811/3190094522', 'heartbeat_back_addr': '172.21.15.3:6815/3190094522', 'heartbeat_front_addr': '172.21.15.3:6813/3190094522', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6816', 'nonce': 1191817370}, {'type': 'v1', 'addr': '172.21.15.118:6817', 'nonce': 1191817370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6818', 'nonce': 1191817370}, {'type': 'v1', 'addr': '172.21.15.118:6819', 'nonce': 1191817370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6822', 'nonce': 1191817370}, {'type': 'v1', 'addr': '172.21.15.118:6823', 'nonce': 1191817370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6820', 'nonce': 1191817370}, {'type': 'v1', 'addr': '172.21.15.118:6821', 'nonce': 1191817370}]}, 'public_addr': '172.21.15.118:6817/1191817370', 'cluster_addr': '172.21.15.118:6819/1191817370', 'heartbeat_back_addr': '172.21.15.118:6823/1191817370', 'heartbeat_front_addr': '172.21.15.118:6821/1191817370', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '6bfe62d5-9c5b-44df-bc4f-cf312837c932', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6816', 'nonce': 3704126567}, {'type': 'v1', 'addr': '172.21.15.177:6817', 'nonce': 3704126567}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6818', 'nonce': 3704126567}, {'type': 'v1', 'addr': '172.21.15.177:6819', 'nonce': 3704126567}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6822', 'nonce': 3704126567}, {'type': 'v1', 'addr': '172.21.15.177:6823', 'nonce': 3704126567}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6820', 'nonce': 3704126567}, {'type': 'v1', 'addr': '172.21.15.177:6821', 'nonce': 3704126567}]}, 'public_addr': '172.21.15.177:6817/3704126567', 'cluster_addr': '172.21.15.177:6819/3704126567', 'heartbeat_back_addr': '172.21.15.177:6823/3704126567', 'heartbeat_front_addr': '172.21.15.177:6821/3704126567', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fe2b6f49-91ab-4692-ab9c-4b3bda3bf429', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6826', 'nonce': 3376854549}, {'type': 'v1', 'addr': '172.21.15.195:6827', 'nonce': 3376854549}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6828', 'nonce': 3376854549}, {'type': 'v1', 'addr': '172.21.15.195:6829', 'nonce': 3376854549}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6832', 'nonce': 3376854549}, {'type': 'v1', 'addr': '172.21.15.195:6833', 'nonce': 3376854549}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6830', 'nonce': 3376854549}, {'type': 'v1', 'addr': '172.21.15.195:6831', 'nonce': 3376854549}]}, 'public_addr': '172.21.15.195:6827/3376854549', 'cluster_addr': '172.21.15.195:6829/3376854549', 'heartbeat_back_addr': '172.21.15.195:6833/3376854549', 'heartbeat_front_addr': '172.21.15.195:6831/3376854549', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '399375fc-ce28-40e2-89b1-e0fc2c1f44ec', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6824', 'nonce': 1858329864}, {'type': 'v1', 'addr': '172.21.15.3:6825', 'nonce': 1858329864}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6826', 'nonce': 1858329864}, {'type': 'v1', 'addr': '172.21.15.3:6827', 'nonce': 1858329864}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6830', 'nonce': 1858329864}, {'type': 'v1', 'addr': '172.21.15.3:6831', 'nonce': 1858329864}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6828', 'nonce': 1858329864}, {'type': 'v1', 'addr': '172.21.15.3:6829', 'nonce': 1858329864}]}, 'public_addr': '172.21.15.3:6825/1858329864', 'cluster_addr': '172.21.15.3:6827/1858329864', 'heartbeat_back_addr': '172.21.15.3:6831/1858329864', 'heartbeat_front_addr': '172.21.15.3:6829/1858329864', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 2650650539}, {'type': 'v1', 'addr': '172.21.15.118:6801', 'nonce': 2650650539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 2650650539}, {'type': 'v1', 'addr': '172.21.15.118:6803', 'nonce': 2650650539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 2650650539}, {'type': 'v1', 'addr': '172.21.15.118:6807', 'nonce': 2650650539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 2650650539}, {'type': 'v1', 'addr': '172.21.15.118:6805', 'nonce': 2650650539}]}, 'public_addr': '172.21.15.118:6801/2650650539', 'cluster_addr': '172.21.15.118:6803/2650650539', 'heartbeat_back_addr': '172.21.15.118:6807/2650650539', 'heartbeat_front_addr': '172.21.15.118:6805/2650650539', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ac6efa66-27ed-494c-956d-22b90be7a8fd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6824', 'nonce': 47176522}, {'type': 'v1', 'addr': '172.21.15.177:6825', 'nonce': 47176522}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6826', 'nonce': 47176522}, {'type': 'v1', 'addr': '172.21.15.177:6827', 'nonce': 47176522}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6830', 'nonce': 47176522}, {'type': 'v1', 'addr': '172.21.15.177:6831', 'nonce': 47176522}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6828', 'nonce': 47176522}, {'type': 'v1', 'addr': '172.21.15.177:6829', 'nonce': 47176522}]}, 'public_addr': '172.21.15.177:6825/47176522', 'cluster_addr': '172.21.15.177:6827/47176522', 'heartbeat_back_addr': '172.21.15.177:6831/47176522', 'heartbeat_front_addr': '172.21.15.177:6829/47176522', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'fb2a81f7-b4f5-4c4c-aff8-801e8b73d192', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6810', 'nonce': 3756385243}, {'type': 'v1', 'addr': '172.21.15.195:6811', 'nonce': 3756385243}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6812', 'nonce': 3756385243}, {'type': 'v1', 'addr': '172.21.15.195:6813', 'nonce': 3756385243}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6816', 'nonce': 3756385243}, {'type': 'v1', 'addr': '172.21.15.195:6817', 'nonce': 3756385243}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6814', 'nonce': 3756385243}, {'type': 'v1', 'addr': '172.21.15.195:6815', 'nonce': 3756385243}]}, 'public_addr': '172.21.15.195:6811/3756385243', 'cluster_addr': '172.21.15.195:6813/3756385243', 'heartbeat_back_addr': '172.21.15.195:6817/3756385243', 'heartbeat_front_addr': '172.21.15.195:6815/3756385243', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': '01aca5fb-932f-4ecb-a9ab-c716a5247373', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 3976393740}, {'type': 'v1', 'addr': '172.21.15.3:6817', 'nonce': 3976393740}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6818', 'nonce': 3976393740}, {'type': 'v1', 'addr': '172.21.15.3:6819', 'nonce': 3976393740}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6822', 'nonce': 3976393740}, {'type': 'v1', 'addr': '172.21.15.3:6823', 'nonce': 3976393740}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6820', 'nonce': 3976393740}, {'type': 'v1', 'addr': '172.21.15.3:6821', 'nonce': 3976393740}]}, 'public_addr': '172.21.15.3:6817/3976393740', 'cluster_addr': '172.21.15.3:6819/3976393740', 'heartbeat_back_addr': '172.21.15.3:6823/3976393740', 'heartbeat_front_addr': '172.21.15.3:6821/3976393740', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': 'd52efa47-5645-4018-b495-ed116da5f7a9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6824', 'nonce': 2321182247}, {'type': 'v1', 'addr': '172.21.15.118:6825', 'nonce': 2321182247}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6826', 'nonce': 2321182247}, {'type': 'v1', 'addr': '172.21.15.118:6827', 'nonce': 2321182247}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6830', 'nonce': 2321182247}, {'type': 'v1', 'addr': '172.21.15.118:6831', 'nonce': 2321182247}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6828', 'nonce': 2321182247}, {'type': 'v1', 'addr': '172.21.15.118:6829', 'nonce': 2321182247}]}, 'public_addr': '172.21.15.118:6825/2321182247', 'cluster_addr': '172.21.15.118:6827/2321182247', 'heartbeat_back_addr': '172.21.15.118:6831/2321182247', 'heartbeat_front_addr': '172.21.15.118:6829/2321182247', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '99005d53-e62d-4c20-8d3c-541fd20cbeb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6808', 'nonce': 3769110015}, {'type': 'v1', 'addr': '172.21.15.177:6809', 'nonce': 3769110015}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6810', 'nonce': 3769110015}, {'type': 'v1', 'addr': '172.21.15.177:6811', 'nonce': 3769110015}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6814', 'nonce': 3769110015}, {'type': 'v1', 'addr': '172.21.15.177:6815', 'nonce': 3769110015}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6812', 'nonce': 3769110015}, {'type': 'v1', 'addr': '172.21.15.177:6813', 'nonce': 3769110015}]}, 'public_addr': '172.21.15.177:6809/3769110015', 'cluster_addr': '172.21.15.177:6811/3769110015', 'heartbeat_back_addr': '172.21.15.177:6815/3769110015', 'heartbeat_front_addr': '172.21.15.177:6813/3769110015', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f460f1c7-0172-4bd9-875f-b34aff85804f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6818', 'nonce': 1553304829}, {'type': 'v1', 'addr': '172.21.15.195:6819', 'nonce': 1553304829}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6820', 'nonce': 1553304829}, {'type': 'v1', 'addr': '172.21.15.195:6821', 'nonce': 1553304829}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6824', 'nonce': 1553304829}, {'type': 'v1', 'addr': '172.21.15.195:6825', 'nonce': 1553304829}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6822', 'nonce': 1553304829}, {'type': 'v1', 'addr': '172.21.15.195:6823', 'nonce': 1553304829}]}, 'public_addr': '172.21.15.195:6819/1553304829', 'cluster_addr': '172.21.15.195:6821/1553304829', 'heartbeat_back_addr': '172.21.15.195:6825/1553304829', 'heartbeat_front_addr': '172.21.15.195:6823/1553304829', 'state': ['exists', 'up']}, {'osd': 12, 'uuid': '7bd5fea1-6e94-478a-8a8f-87459b4030a2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6800', 'nonce': 3221525577}, {'type': 'v1', 'addr': '172.21.15.3:6801', 'nonce': 3221525577}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 3221525577}, {'type': 'v1', 'addr': '172.21.15.3:6803', 'nonce': 3221525577}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 3221525577}, {'type': 'v1', 'addr': '172.21.15.3:6807', 'nonce': 3221525577}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 3221525577}, {'type': 'v1', 'addr': '172.21.15.3:6805', 'nonce': 3221525577}]}, 'public_addr': '172.21.15.3:6801/3221525577', 'cluster_addr': '172.21.15.3:6803/3221525577', 'heartbeat_back_addr': '172.21.15.3:6807/3221525577', 'heartbeat_front_addr': '172.21.15.3:6805/3221525577', 'state': ['exists', 'up']}, {'osd': 13, 'uuid': '89c43816-fffc-42a6-98af-7f215631d595', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 22, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 84283894}, {'type': 'v1', 'addr': '172.21.15.118:6809', 'nonce': 84283894}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 84283894}, {'type': 'v1', 'addr': '172.21.15.118:6811', 'nonce': 84283894}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 84283894}, {'type': 'v1', 'addr': '172.21.15.118:6815', 'nonce': 84283894}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 84283894}, {'type': 'v1', 'addr': '172.21.15.118:6813', 'nonce': 84283894}]}, 'public_addr': '172.21.15.118:6809/84283894', 'cluster_addr': '172.21.15.118:6811/84283894', 'heartbeat_back_addr': '172.21.15.118:6815/84283894', 'heartbeat_front_addr': '172.21.15.118:6813/84283894', 'state': ['exists', 'up']}, {'osd': 14, 'uuid': '33cb3383-2d68-4928-be43-cfc215f6b080', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6800', 'nonce': 1752008388}, {'type': 'v1', 'addr': '172.21.15.177:6801', 'nonce': 1752008388}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6802', 'nonce': 1752008388}, {'type': 'v1', 'addr': '172.21.15.177:6803', 'nonce': 1752008388}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6806', 'nonce': 1752008388}, {'type': 'v1', 'addr': '172.21.15.177:6807', 'nonce': 1752008388}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.177:6804', 'nonce': 1752008388}, {'type': 'v1', 'addr': '172.21.15.177:6805', 'nonce': 1752008388}]}, 'public_addr': '172.21.15.177:6801/1752008388', 'cluster_addr': '172.21.15.177:6803/1752008388', 'heartbeat_back_addr': '172.21.15.177:6807/1752008388', 'heartbeat_front_addr': '172.21.15.177:6805/1752008388', 'state': ['exists', 'up']}, {'osd': 15, 'uuid': 'b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 21, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6802', 'nonce': 3564981919}, {'type': 'v1', 'addr': '172.21.15.195:6803', 'nonce': 3564981919}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6804', 'nonce': 3564981919}, {'type': 'v1', 'addr': '172.21.15.195:6805', 'nonce': 3564981919}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6808', 'nonce': 3564981919}, {'type': 'v1', 'addr': '172.21.15.195:6809', 'nonce': 3564981919}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.195:6806', 'nonce': 3564981919}, {'type': 'v1', 'addr': '172.21.15.195:6807', 'nonce': 3564981919}]}, 'public_addr': '172.21.15.195:6803/3564981919', 'cluster_addr': '172.21.15.195:6805/3564981919', 'heartbeat_back_addr': '172.21.15.195:6809/3564981919', 'heartbeat_front_addr': '172.21.15.195:6807/3564981919', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.390713+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.329190+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.172914+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.553892+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.795237+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:01.952515+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.696414+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.100374+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.720464+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.793071+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:01.982210+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.697339+0000', 'dead_epoch': 0}, {'osd': 12, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.478801+0000', 'dead_epoch': 0}, {'osd': 13, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:02.343858+0000', 'dead_epoch': 0}, {'osd': 14, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:01.923086+0000', 'dead_epoch': 0}, {'osd': 15, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-16T02:14:01.888880+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}, 'shecprofile': {'c': '2', 'crush-device-class': '', 'crush-failure-domain': 'osd', 'crush-num-failure-domains': '0', 'crush-osds-per-failure-domain': '0', 'crush-root': 'default', 'k': '4', 'm': '3', 'name': 'shecprofile', 'plugin': 'shec', 'technique': 'multiple'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 7, 'length': 1}, {'begin': 9, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 5}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-16T02:14:41.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:14:41.897 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:41.898 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:41.898 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:41.898 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:41.899 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:41.899 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:41.899 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:41.899 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:41.900 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:41.890+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:41.910 INFO:tasks.rados.rados.0.smithi195.stdout:161: done (6 left) 2024-09-16T02:14:41.910 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 39 current snap is 10 2024-09-16T02:14:41.910 INFO:tasks.rados.rados.0.smithi195.stdout:162: seq_num 83 ranges {638976=638976,1277952=622592,1900544=147456} 2024-09-16T02:14:41.917 INFO:tasks.rados.rados.0.smithi195.stdout:162: writing smithi19530025-39 from 638976 to 1277952 tid 1 2024-09-16T02:14:41.922 INFO:tasks.rados.rados.0.smithi195.stdout:162: writing smithi19530025-39 from 1277952 to 1900544 tid 2 2024-09-16T02:14:41.923 INFO:tasks.rados.rados.0.smithi195.stdout:162: writing smithi19530025-39 from 1900544 to 2048000 tid 3 2024-09-16T02:14:41.923 INFO:tasks.rados.rados.0.smithi195.stdout:163: snap_remove snap 6 2024-09-16T02:14:41.924 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:41.924 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:41.960 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:41.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:41.954+0000 7f27c9ac0640 0 -- 172.21.15.3:0/725939395 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f27c4224f50 msgr2=0x7f27c4227340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:41.996 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:41.990+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.031 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.749069659, 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.032 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:14:42.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.748876884000001, 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.038 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:42.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:14:42.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.044 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.748818840999999, 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:14:42.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:14:42.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:52 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.719951277, 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010472922000000001, 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 52 2024-09-16T02:14:42.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308356+0000", 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3439999999999999e-06 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308602+0000", 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024608900000000001 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308617+0000", 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5695000000000001e-05 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308627+0000", 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0759999999999992e-06 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308714+0000", 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7399999999999997e-05 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308908+0000", 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019356000000000001 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.310117+0000", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001209679 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312215+0000", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020981490000000001 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318705+0000", 2024-09-16T02:14:42.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064893049999999999 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318826+0000", 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121625 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:42.057 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.719661910999999, 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:42.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:14:42.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.719608852, 2024-09-16T02:14:42.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:42.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:14:42.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:14:42.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=9, reqid=client.4598.0:73, at_version=35'7, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.690359517000001, 2024-09-16T02:14:42.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075239130000000001, 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337945+0000", 2024-09-16T02:14:42.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.337946+0000", 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.338051+0000", 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000104225 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343980+0000", 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059297350000000002 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343990+0000", 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4040000000000001e-06 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.344047+0000", 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7324000000000002e-05 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.344058+0000", 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0813e-05 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.345433+0000", 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013752059999999999 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.345469+0000", 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5880000000000002e-05 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:42.074 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.684427389, 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:42.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:14:42.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:14:42.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.079 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s4 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6709167640000002, 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083712849999999991, 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:42.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357389+0000", 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.477e-06 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357534+0000", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000144795 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357646+0000", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011166 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357668+0000", 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1950000000000002e-05 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357901+0000", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023326999999999999 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357931+0000", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9954000000000002e-05 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365676+0000", 2024-09-16T02:14:42.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007745282 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365759+0000", 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2897000000000003e-05 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.0s6 45/32 ECSubWrite(tid=12, reqid=client.4598.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=45'3))", 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:42.084 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6707727869999998, 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081701599999999992, 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:42.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357534+0000", 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0870000000000001e-06 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357564+0000", 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0195000000000001e-05 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357660+0000", 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5581000000000005e-05 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357682+0000", 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2688999999999999e-05 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358020+0000", 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033732899999999998 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6959000000000001e-05 2024-09-16T02:14:42.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365601+0000", 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075544330000000002 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365702+0000", 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100887 2024-09-16T02:14:42.089 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs6 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.358014+0000", 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6702902069999999, 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077544019999999996, 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358014+0000", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358014+0000", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358015+0000", 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9999999999999997e-07 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358065+0000", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9926999999999998e-05 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358170+0000", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010543999999999999 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358190+0000", 2024-09-16T02:14:42.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9805000000000001e-05 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358324+0000", 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013364099999999999 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358345+0000", 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1194000000000001e-05 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365706+0000", 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0073607569999999999 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365769+0000", 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3038000000000002e-05 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.9s2 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:42.094 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.359951+0000", 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6683534570000003, 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060006720000000003, 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359951+0000", 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359951+0000", 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359953+0000", 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.677e-06 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359977+0000", 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4536999999999999e-05 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360064+0000", 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7016999999999997e-05 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360088+0000", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3308000000000001e-05 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360240+0000", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015231500000000001 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360263+0000", 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3147999999999999e-05 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365836+0000", 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055727340000000002 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365952+0000", 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115936 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6587844460000003, 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010210629000000001, 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:42.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369521+0000", 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2500000000000005e-07 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369567+0000", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6436000000000001e-05 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369723+0000", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015562700000000001 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369744+0000", 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1522000000000002e-05 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369953+0000", 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020849599999999999 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369975+0000", 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2484999999999999e-05 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379641+0000", 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096656490000000001 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379731+0000", 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9789000000000004e-05 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=19, reqid=osd.8.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:42.104 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.6568964340000001, 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094212150000000001, 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371410+0000", 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371440+0000", 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9739000000000001e-05 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371544+0000", 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103787 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371566+0000", 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2673000000000001e-05 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371732+0000", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016536799999999999 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371756+0000", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4059000000000001e-05 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380738+0000", 2024-09-16T02:14:42.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0089818060000000002 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380829+0000", 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1782999999999999e-05 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.6712146910000003, 2024-09-16T02:14:42.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:14:42.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.115 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.6711054540000001, 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.116 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:14:42.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.121841273, 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.123 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:14:42.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:14:42.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:14:42.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.121725546, 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.130 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:14:42.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.135 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.1001907630000001, 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:14:42.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.138 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.139 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:14:42.140 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 53/32 ECSubWrite(tid=29, reqid=client.4598.0:421, at_version=53'20, trim_to=0'0, roll_forward_to=48'18))", 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:41.925828+0000", 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.102476435, 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055202439999999997, 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:42.141 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.925828+0000", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.925828+0000", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.925829+0000", 2024-09-16T02:14:42.142 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1800000000000002e-07 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.927698+0000", 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001869007 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.927743+0000", 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5515e-05 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:42.143 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.927759+0000", 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6041000000000001e-05 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.927838+0000", 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8987999999999997e-05 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.927848+0000", 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8290000000000003e-06 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.144 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.931309+0000", 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034615539999999999 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.931348+0000", 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8992000000000002e-05 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:42.145 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:42.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:42.147 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:42.142+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:42.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:42.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:42.248 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:42.247+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:42.253 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:14:42.254 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:14:42.274 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":57,"stamp":"2024-09-16T02:14:42.002402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":40976947,"num_objects":69,"num_object_clones":19,"num_object_copies":463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":1,"num_read":313,"num_read_kb":34202,"num_write":456,"num_write_kb":50322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":470,"ondisk_log_size":470,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":16,"kb":1509949440,"kb_used":526620,"kb_used_data":77492,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509422820,"statfs":{"total":1546188226560,"available":1545648967680,"internally_reserved":0,"allocated":79351808,"data_stored":90127912,"data_compressed":1328182,"data_compressed_allocated":1859584,"data_compressed_original":4579328,"omap_allocated":25522,"internal_metadata":459512910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16252928,"num_objects":19,"num_object_clones":17,"num_object_copies":133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":73,"num_read_kb":27300,"num_write":122,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.622389"},"pg_stats":[{"pgid":"3.f","version":"48'21","reported_seq":123,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.393082+0000","last_change":"2024-09-16T02:14:37.379834+0000","last_active":"2024-09-16T02:14:39.393082+0000","last_peered":"2024-09-16T02:14:39.393082+0000","last_clean":"2024-09-16T02:14:39.393082+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:14:39.393082+0000","last_undegraded":"2024-09-16T02:14:39.393082+0000","last_fullsized":"2024-09-16T02:14:39.393082+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:37.493264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001648095,"stat_sum":{"num_bytes":2457600,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":2663,"num_write":11,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.e","version":"48'18","reported_seq":112,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.492970+0000","last_change":"2024-09-16T02:14:37.379302+0000","last_active":"2024-09-16T02:14:38.492970+0000","last_peered":"2024-09-16T02:14:38.492970+0000","last_clean":"2024-09-16T02:14:38.492970+0000","last_became_active":"2024-09-16T02:14:22.491985+0000","last_became_peered":"2024-09-16T02:14:22.491985+0000","last_unstale":"2024-09-16T02:14:38.492970+0000","last_undegraded":"2024-09-16T02:14:38.492970+0000","last_fullsized":"2024-09-16T02:14:38.492970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.690563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00114401,"stat_sum":{"num_bytes":3211264,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":1877,"num_write":13,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,10,11,0,15],"acting":[8,9,1,10,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.d","version":"49'30","reported_seq":119,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.518684+0000","last_change":"2024-09-16T02:14:37.379312+0000","last_active":"2024-09-16T02:14:38.518684+0000","last_peered":"2024-09-16T02:14:38.518684+0000","last_clean":"2024-09-16T02:14:38.518684+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:14:38.518684+0000","last_undegraded":"2024-09-16T02:14:38.518684+0000","last_fullsized":"2024-09-16T02:14:38.518684+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.984921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008417796,"stat_sum":{"num_bytes":3801088,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":36,"num_write_kb":3712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.c","version":"48'13","reported_seq":100,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.411839+0000","last_change":"2024-09-16T02:14:37.396758+0000","last_active":"2024-09-16T02:14:38.411839+0000","last_peered":"2024-09-16T02:14:38.411839+0000","last_clean":"2024-09-16T02:14:38.411839+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:14:38.411839+0000","last_undegraded":"2024-09-16T02:14:38.411839+0000","last_fullsized":"2024-09-16T02:14:38.411839+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.332390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095921799999999996,"stat_sum":{"num_bytes":1523712,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":1488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.b","version":"35'6","reported_seq":97,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.393334+0000","last_change":"2024-09-16T02:14:37.380221+0000","last_active":"2024-09-16T02:14:38.393334+0000","last_peered":"2024-09-16T02:14:38.393334+0000","last_clean":"2024-09-16T02:14:38.393334+0000","last_became_active":"2024-09-16T02:14:22.493029+0000","last_became_peered":"2024-09-16T02:14:22.493029+0000","last_unstale":"2024-09-16T02:14:38.393334+0000","last_undegraded":"2024-09-16T02:14:38.393334+0000","last_fullsized":"2024-09-16T02:14:38.393334+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:36.636971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001987628,"stat_sum":{"num_bytes":1048576,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":1297,"num_write":6,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,10,13],"acting":[7,15,1,9,8,10,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.a","version":"0'0","reported_seq":87,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.408497+0000","last_change":"2024-09-16T02:14:37.372050+0000","last_active":"2024-09-16T02:14:38.408497+0000","last_peered":"2024-09-16T02:14:38.408497+0000","last_clean":"2024-09-16T02:14:38.408497+0000","last_became_active":"2024-09-16T02:14:22.348067+0000","last_became_peered":"2024-09-16T02:14:22.348067+0000","last_unstale":"2024-09-16T02:14:38.408497+0000","last_undegraded":"2024-09-16T02:14:38.408497+0000","last_fullsized":"2024-09-16T02:14:38.408497+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:24.989060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013299360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,1,14,8,15,2],"acting":[6,10,1,14,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.9","version":"45'6","reported_seq":96,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.394556+0000","last_change":"2024-09-16T02:14:37.376804+0000","last_active":"2024-09-16T02:14:39.394556+0000","last_peered":"2024-09-16T02:14:39.394556+0000","last_clean":"2024-09-16T02:14:39.394556+0000","last_became_active":"2024-09-16T02:14:22.482031+0000","last_became_peered":"2024-09-16T02:14:22.482031+0000","last_unstale":"2024-09-16T02:14:39.394556+0000","last_undegraded":"2024-09-16T02:14:39.394556+0000","last_fullsized":"2024-09-16T02:14:39.394556+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:24.996372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078738799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":3,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,1,15,4,14,11,2],"acting":[12,1,15,4,14,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.8","version":"49'31","reported_seq":132,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.394181+0000","last_change":"2024-09-16T02:14:37.371437+0000","last_active":"2024-09-16T02:14:39.394181+0000","last_peered":"2024-09-16T02:14:39.394181+0000","last_clean":"2024-09-16T02:14:39.394181+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:14:39.394181+0000","last_undegraded":"2024-09-16T02:14:39.394181+0000","last_fullsized":"2024-09-16T02:14:39.394181+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:42.095052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067039200000000001,"stat_sum":{"num_bytes":3424256,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":1307,"num_write":39,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"1.0","version":"26'112","reported_seq":196,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.910522+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:14:39.910522+0000","last_peered":"2024-09-16T02:14:39.910522+0000","last_clean":"2024-09-16T02:14:39.910522+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:14:39.910522+0000","last_undegraded":"2024-09-16T02:14:39.910522+0000","last_fullsized":"2024-09-16T02:14:39.910522+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.805198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":65,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.394173+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:14:39.394173+0000","last_peered":"2024-09-16T02:14:39.394173+0000","last_clean":"2024-09-16T02:14:39.394173+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:14:39.394173+0000","last_undegraded":"2024-09-16T02:14:39.394173+0000","last_fullsized":"2024-09-16T02:14:39.394173+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:34.654666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"49'43","reported_seq":157,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.910743+0000","last_change":"2024-09-16T02:14:37.379013+0000","last_active":"2024-09-16T02:14:39.910743+0000","last_peered":"2024-09-16T02:14:39.910743+0000","last_clean":"2024-09-16T02:14:39.910743+0000","last_became_active":"2024-09-16T02:14:22.476669+0000","last_became_peered":"2024-09-16T02:14:22.476669+0000","last_unstale":"2024-09-16T02:14:39.910743+0000","last_undegraded":"2024-09-16T02:14:39.910743+0000","last_fullsized":"2024-09-16T02:14:39.910743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:37.518559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010434929999999999,"stat_sum":{"num_bytes":6881280,"num_objects":10,"num_object_clones":3,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":6254,"num_write":28,"num_write_kb":7296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,10,14,1,0],"acting":[13,12,6,10,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.0","version":"0'0","reported_seq":63,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.392609+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:14:39.392609+0000","last_peered":"2024-09-16T02:14:39.392609+0000","last_clean":"2024-09-16T02:14:39.392609+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:14:39.392609+0000","last_undegraded":"2024-09-16T02:14:39.392609+0000","last_fullsized":"2024-09-16T02:14:39.392609+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:25.489222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"49'78","reported_seq":167,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.423202+0000","last_change":"2024-09-16T02:14:37.622653+0000","last_active":"2024-09-16T02:14:38.423202+0000","last_peered":"2024-09-16T02:14:38.423202+0000","last_clean":"2024-09-16T02:14:38.423202+0000","last_became_active":"2024-09-16T02:14:22.493319+0000","last_became_peered":"2024-09-16T02:14:22.493319+0000","last_unstale":"2024-09-16T02:14:38.423202+0000","last_undegraded":"2024-09-16T02:14:38.423202+0000","last_fullsized":"2024-09-16T02:14:38.423202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:31.677152+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.085934152,"stat_sum":{"num_bytes":9076736,"num_objects":12,"num_object_clones":5,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":26,"num_read_kb":4226,"num_write":80,"num_write_kb":9936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.910471+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:14:39.910471+0000","last_peered":"2024-09-16T02:14:39.910471+0000","last_clean":"2024-09-16T02:14:39.910471+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:14:39.910471+0000","last_undegraded":"2024-09-16T02:14:39.910471+0000","last_fullsized":"2024-09-16T02:14:39.910471+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:19.434059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"48'8","reported_seq":100,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.394498+0000","last_change":"2024-09-16T02:14:37.376636+0000","last_active":"2024-09-16T02:14:39.394498+0000","last_peered":"2024-09-16T02:14:39.394498+0000","last_clean":"2024-09-16T02:14:39.394498+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:14:39.394498+0000","last_undegraded":"2024-09-16T02:14:39.394498+0000","last_fullsized":"2024-09-16T02:14:39.394498+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:37.248759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078447399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.2","version":"28'2","reported_seq":65,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.392751+0000","last_change":"2024-09-16T02:14:10.180900+0000","last_active":"2024-09-16T02:14:39.392751+0000","last_peered":"2024-09-16T02:14:39.392751+0000","last_clean":"2024-09-16T02:14:39.392751+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:14:39.392751+0000","last_undegraded":"2024-09-16T02:14:39.392751+0000","last_fullsized":"2024-09-16T02:14:39.392751+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.697910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"45'20","reported_seq":122,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.910791+0000","last_change":"2024-09-16T02:14:37.378710+0000","last_active":"2024-09-16T02:14:39.910791+0000","last_peered":"2024-09-16T02:14:39.910791+0000","last_clean":"2024-09-16T02:14:39.910791+0000","last_became_active":"2024-09-16T02:14:22.484621+0000","last_became_peered":"2024-09-16T02:14:22.484621+0000","last_unstale":"2024-09-16T02:14:39.910791+0000","last_undegraded":"2024-09-16T02:14:39.910791+0000","last_fullsized":"2024-09-16T02:14:39.910791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:21.431729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012106249999999999,"stat_sum":{"num_bytes":983040,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":5328,"num_write":8,"num_write_kb":1392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,10],"acting":[13,2,11,14,4,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"3.5","version":"45'16","reported_seq":121,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-16T02:14:39.944124+0000","last_change":"2024-09-16T02:14:37.392598+0000","last_active":"2024-09-16T02:14:39.944124+0000","last_peered":"2024-09-16T02:14:39.944124+0000","last_clean":"2024-09-16T02:14:39.944124+0000","last_became_active":"2024-09-16T02:14:22.476388+0000","last_became_peered":"2024-09-16T02:14:22.476388+0000","last_unstale":"2024-09-16T02:14:39.944124+0000","last_undegraded":"2024-09-16T02:14:39.944124+0000","last_fullsized":"2024-09-16T02:14:39.944124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.137965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025166849999999998,"stat_sum":{"num_bytes":1654784,"num_objects":3,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":8682,"num_write":13,"num_write_kb":2480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,15,14,0,7,8],"acting":[10,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":59,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.393502+0000","last_change":"2024-09-16T02:14:10.237875+0000","last_active":"2024-09-16T02:14:38.393502+0000","last_peered":"2024-09-16T02:14:38.393502+0000","last_clean":"2024-09-16T02:14:38.393502+0000","last_became_active":"2024-09-16T02:14:08.168758+0000","last_became_peered":"2024-09-16T02:14:08.168758+0000","last_unstale":"2024-09-16T02:14:38.393502+0000","last_undegraded":"2024-09-16T02:14:38.393502+0000","last_fullsized":"2024-09-16T02:14:38.393502+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:13.210132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084871600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"48'12","reported_seq":102,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.393932+0000","last_change":"2024-09-16T02:14:37.381346+0000","last_active":"2024-09-16T02:14:38.393932+0000","last_peered":"2024-09-16T02:14:38.393932+0000","last_clean":"2024-09-16T02:14:38.393932+0000","last_became_active":"2024-09-16T02:14:22.346842+0000","last_became_peered":"2024-09-16T02:14:22.346842+0000","last_unstale":"2024-09-16T02:14:38.393932+0000","last_undegraded":"2024-09-16T02:14:38.393932+0000","last_fullsized":"2024-09-16T02:14:38.393932+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:28.503374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001962905,"stat_sum":{"num_bytes":1458176,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":1356,"num_write":27,"num_write_kb":1472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.415302+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:14:38.415302+0000","last_peered":"2024-09-16T02:14:38.415302+0000","last_clean":"2024-09-16T02:14:38.415302+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:14:38.415302+0000","last_undegraded":"2024-09-16T02:14:38.415302+0000","last_fullsized":"2024-09-16T02:14:38.415302+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:18.037298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"48'41","reported_seq":132,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.411012+0000","last_change":"2024-09-16T02:14:37.377282+0000","last_active":"2024-09-16T02:14:38.411012+0000","last_peered":"2024-09-16T02:14:38.411012+0000","last_clean":"2024-09-16T02:14:38.411012+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:14:38.411012+0000","last_undegraded":"2024-09-16T02:14:38.411012+0000","last_fullsized":"2024-09-16T02:14:38.411012+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:33.181692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099272399999999995,"stat_sum":{"num_bytes":4472832,"num_objects":8,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":7,"num_write":23,"num_write_kb":5440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.6","version":"0'0","reported_seq":62,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.394866+0000","last_change":"2024-09-16T02:14:10.173168+0000","last_active":"2024-09-16T02:14:38.394866+0000","last_peered":"2024-09-16T02:14:38.394866+0000","last_clean":"2024-09-16T02:14:38.394866+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:14:38.394866+0000","last_undegraded":"2024-09-16T02:14:38.394866+0000","last_fullsized":"2024-09-16T02:14:38.394866+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:26.305109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"49'12","reported_seq":106,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.394572+0000","last_change":"2024-09-16T02:14:37.383297+0000","last_active":"2024-09-16T02:14:38.394572+0000","last_peered":"2024-09-16T02:14:38.394572+0000","last_clean":"2024-09-16T02:14:38.394572+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:14:38.394572+0000","last_undegraded":"2024-09-16T02:14:38.394572+0000","last_fullsized":"2024-09-16T02:14:38.394572+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:32.477271+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012553343,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":13,"num_read_kb":1122,"num_write":7,"num_write_kb":384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-16T02:14:38.408398+0000","last_change":"2024-09-16T02:14:10.397195+0000","last_active":"2024-09-16T02:14:38.408398+0000","last_peered":"2024-09-16T02:14:38.408398+0000","last_clean":"2024-09-16T02:14:38.408398+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:14:38.408398+0000","last_undegraded":"2024-09-16T02:14:38.408398+0000","last_fullsized":"2024-09-16T02:14:38.408398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:27.053981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":40386560,"num_objects":65,"num_object_clones":19,"num_object_copies":455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":1,"num_read":227,"num_read_kb":34129,"num_write":309,"num_write_kb":47648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":70676480,"data_stored":83693568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":355,"ondisk_log_size":355,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":9},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34996,"kb_used_data":3808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336844,"statfs":{"total":96636764160,"available":96600928256,"internally_reserved":0,"allocated":3899392,"data_stored":4472252,"data_compressed":111109,"data_compressed_allocated":151552,"data_compressed_original":733184,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":4468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":4575232,"data_stored":4899741,"data_compressed":81514,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,4,5,6,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280523,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35308,"kb_used_data":4116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336532,"statfs":{"total":96636764160,"available":96600608768,"internally_reserved":0,"allocated":4214784,"data_stored":5254044,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313227,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33556,"kb_used_data":6524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338284,"statfs":{"total":96636764160,"available":96602402816,"internally_reserved":0,"allocated":6680576,"data_stored":7789468,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280523,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31556,"kb_used_data":4528,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340284,"statfs":{"total":96636764160,"available":96604450816,"internally_reserved":0,"allocated":4636672,"data_stored":4860828,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313225,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32676,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339164,"statfs":{"total":96636764160,"available":96603303936,"internally_reserved":0,"allocated":5779456,"data_stored":6282140,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280522,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34776,"kb_used_data":3580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337064,"statfs":{"total":96636764160,"available":96601153536,"internally_reserved":0,"allocated":3665920,"data_stored":3808175,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31936,"kb_used_data":4904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339904,"statfs":{"total":96636764160,"available":96604061696,"internally_reserved":0,"allocated":5021696,"data_stored":5950364,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280521,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32400,"kb_used_data":5368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339440,"statfs":{"total":96636764160,"available":96603586560,"internally_reserved":0,"allocated":5496832,"data_stored":5884847,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280522,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30180,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341660,"statfs":{"total":96636764160,"available":96605859840,"internally_reserved":0,"allocated":3227648,"data_stored":3414940,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280522,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34452,"kb_used_data":7428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94337388,"statfs":{"total":96636764160,"available":96601485312,"internally_reserved":0,"allocated":7606272,"data_stored":9305532,"data_compressed":111109,"data_compressed_allocated":151552,"data_compressed_original":733184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33160,"kb_used_data":6132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94338680,"statfs":{"total":96636764160,"available":96602808320,"internally_reserved":0,"allocated":6279168,"data_stored":7519132,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,7,8,9,10,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29596,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342244,"statfs":{"total":96636764160,"available":96606457856,"internally_reserved":0,"allocated":2629632,"data_stored":2608028,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,5,6,7,8,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280522,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37604,"kb_used_data":6412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94334236,"statfs":{"total":96636764160,"available":96598257664,"internally_reserved":0,"allocated":6565888,"data_stored":7748508,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280522,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31992,"kb_used_data":4964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94339848,"statfs":{"total":96636764160,"available":96604004352,"internally_reserved":0,"allocated":5083136,"data_stored":6212508,"data_compressed":78578,"data_compressed_allocated":110592,"data_compressed_original":221184,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280522,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":30932,"kb_used_data":3896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340908,"statfs":{"total":96636764160,"available":96605089792,"internally_reserved":0,"allocated":3989504,"data_stored":4117405,"data_compressed":81514,"data_compressed_allocated":114688,"data_compressed_original":229376,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7012352,"data_stored":8441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4104192,"data_stored":4526080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3129344,"data_stored":3473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5746688,"data_stored":7184384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2273280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4489216,"data_stored":5615616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6033408,"data_stored":7413760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4960256,"data_stored":5550080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":2695168,"data_stored":3080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4550656,"data_stored":5877760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":3444736,"data_stored":3772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3682304,"data_stored":4919296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4030464,"data_stored":4554752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":3305472,"data_stored":3608576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":6148096,"data_stored":7454720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":5246976,"data_stored":5947392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:14:42.276 INFO:tasks.thrashosds.thrasher:Setting 3.9 to [7, 13, 1, 0, 8, 6, 12] 2024-09-16T02:14:42.276 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.9', '7', '13', '1', '0', '8', '6', '12'] 2024-09-16T02:14:42.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.9 7 13 1 0 8 6 12 2024-09-16T02:14:42.284 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:42.284 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:42.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:42.349 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:42.346+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:42.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:14:42.466 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:42.466 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:42.467 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:42.462+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:42.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:42.551 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:42.550+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:42.647 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:42.647 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:42.652 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:42.647+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:42.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:42.752 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:42.746+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:42.771 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:42.772 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:42.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:42.853 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:42.851+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:42.909 INFO:teuthology.orchestra.run.smithi003.stderr:set 3.9 pg_upmap mapping to [7,13,1,0,8,6,12] 2024-09-16T02:14:42.909 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:163: done (7 left) 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 5 current snap is 10 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:156: finishing write tid 5 to smithi19530025-8 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:156: oid 8 updating version 0 to 20 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:156: oid 8 version 20 is already newer than 18 2024-09-16T02:14:42.915 INFO:tasks.rados.rados.0.smithi195.stdout:156: oid 8 updating version 20 to 21 2024-09-16T02:14:42.916 INFO:tasks.rados.rados.0.smithi195.stdout:156: oid 8 version 21 is already newer than 19 2024-09-16T02:14:42.916 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 8 v 21 (ObjNum 80 snap 10 seq_num 80) dirty exists 2024-09-16T02:14:42.916 INFO:tasks.rados.rados.0.smithi195.stdout:156: left oid 8 (ObjNum 80 snap 10 seq_num 80) 2024-09-16T02:14:42.916 INFO:tasks.rados.rados.0.smithi195.stdout:158: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-16T02:14:42.916 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:42.940 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:42.954 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:42.954+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:42.963 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:42.963 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:42.963 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 60391 2024-09-16T02:14:42.974 INFO:tasks.rados.rados.0.smithi195.stdout:157: finishing write tid 1 to smithi19530025-47 2024-09-16T02:14:42.975 INFO:tasks.rados.rados.0.smithi195.stdout:157: finishing write tid 2 to smithi19530025-47 2024-09-16T02:14:42.975 INFO:tasks.rados.rados.0.smithi195.stdout:164: seq_num 84 ranges {81920=655360,737280=638976,1376256=131072} 2024-09-16T02:14:42.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:42.987 INFO:tasks.rados.rados.0.smithi195.stdout:164: writing smithi19530025-5 from 81920 to 737280 tid 1 2024-09-16T02:14:42.996 INFO:tasks.rados.rados.0.smithi195.stdout:164: writing smithi19530025-5 from 737280 to 1376256 tid 2 2024-09-16T02:14:42.999 INFO:tasks.rados.rados.0.smithi195.stdout:164: writing smithi19530025-5 from 1376256 to 1507328 tid 3 2024-09-16T02:14:42.999 INFO:tasks.rados.rados.0.smithi195.stdout:156: done (7 left) 2024-09-16T02:14:42.999 INFO:tasks.rados.rados.0.smithi195.stdout:158: done (6 left) 2024-09-16T02:14:42.999 INFO:tasks.rados.rados.0.smithi195.stdout:165: rollback oid 42 current snap is 10 2024-09-16T02:14:42.999 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 42 to 8 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:157: finishing write tid 3 to smithi19530025-47 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: finishing write tid 1 to smithi19530025-19 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: finishing write tid 2 to smithi19530025-19 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: finishing write tid 3 to smithi19530025-19 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: finishing write tid 4 to smithi19530025-19 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: finishing write tid 5 to smithi19530025-19 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: oid 19 updating version 0 to 48 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: oid 19 updating version 48 to 49 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: oid 19 updating version 49 to 50 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: oid 19 version 50 is already newer than 47 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 19 v 50 (ObjNum 82 snap 10 seq_num 82) dirty exists 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:159: left oid 19 (ObjNum 82 snap 10 seq_num 82) 2024-09-16T02:14:43.000 INFO:tasks.rados.rados.0.smithi195.stdout:157: finishing write tid 4 to smithi19530025-47 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: finishing write tid 5 to smithi19530025-47 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: oid 47 updating version 0 to 9 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: oid 47 version 9 is already newer than 8 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: oid 47 updating version 9 to 10 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: oid 47 version 10 is already newer than 7 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 47 v 10 (ObjNum 81 snap 10 seq_num 81) dirty exists 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:157: left oid 47 (ObjNum 81 snap 10 seq_num 81) 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:155: finishing copy_from to smithi19530025-18 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 18 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:162: finishing write tid 1 to smithi19530025-39 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:162: finishing write tid 2 to smithi19530025-39 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:162: finishing write tid 3 to smithi19530025-39 2024-09-16T02:14:43.001 INFO:tasks.rados.rados.0.smithi195.stdout:162: finishing write tid 4 to smithi19530025-39 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: finishing write tid 5 to smithi19530025-39 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: oid 39 updating version 0 to 20 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: oid 39 updating version 20 to 23 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: oid 39 version 23 is already newer than 22 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: oid 39 version 23 is already newer than 21 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 39 v 23 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: left oid 39 (ObjNum 83 snap 10 seq_num 83) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:164: finishing write tid 1 to smithi19530025-5 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:155: done (6 left) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:157: done (5 left) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:159: done (4 left) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:160: done (3 left) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:162: done (2 left) 2024-09-16T02:14:43.002 INFO:tasks.rados.rados.0.smithi195.stdout:166: delete oid 28 current snap is 10 2024-09-16T02:14:43.008 INFO:tasks.rados.rados.0.smithi195.stdout:165: finishing rollback tid 0 to smithi19530025-42 2024-09-16T02:14:43.010 INFO:tasks.rados.rados.0.smithi195.stdout:165: finishing rollback tid 1 to smithi19530025-42 2024-09-16T02:14:43.010 INFO:tasks.rados.rados.0.smithi195.stdout:166: done (2 left) 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:167: read oid 16 snap 8 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:167: expect deleted 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:168: rollback oid 35 current snap is 10 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 35 to 8 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:169: read oid 11 snap -1 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:169: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:170: rollback oid 31 current snap is 10 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 31 to 8 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:171: read oid 6 snap -1 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:171: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:14:43.011 INFO:tasks.rados.rados.0.smithi195.stdout:172: setattr oid 20 current snap is 10 2024-09-16T02:14:43.013 INFO:tasks.rados.rados.0.smithi195.stdout:173: snap_create 2024-09-16T02:14:43.025 INFO:tasks.rados.rados.0.smithi195.stdout:171: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:14:43.026 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 84356 2024-09-16T02:14:43.027 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 84747 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 52819 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:164: finishing write tid 2 to smithi19530025-5 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:164: finishing write tid 3 to smithi19530025-5 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:165: finishing rollback tid 2 to smithi19530025-42 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 42 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:170: finishing rollback tid 0 to smithi19530025-31 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:169: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:14:43.031 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:43.055 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:43.054+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:43.069 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1536068 2024-09-16T02:14:43.081 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1196682 2024-09-16T02:14:43.088 INFO:tasks.rados.rados.0.smithi195.stdout:168: finishing rollback tid 0 to smithi19530025-35 2024-09-16T02:14:43.088 INFO:tasks.rados.rados.0.smithi195.stdout:168: finishing rollback tid 1 to smithi19530025-35 2024-09-16T02:14:43.088 INFO:tasks.rados.rados.0.smithi195.stdout:170: finishing rollback tid 1 to smithi19530025-31 2024-09-16T02:14:43.132 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.132 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.145 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:43.156 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:43.154+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:43.170 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 20 v 60 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-16T02:14:43.257 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:43.250+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:43.261 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.261 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:43.358 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:43.351+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:43.412 INFO:tasks.rados.rados.0.smithi195.stdout:168: finishing rollback tid 2 to smithi19530025-35 2024-09-16T02:14:43.412 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 35 v 53 (ObjNum 65 snap 7 seq_num 65) dirty exists 2024-09-16T02:14:43.444 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.444 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:43.459 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:43.455+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:43.560 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:43.558+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:43.581 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.581 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:43.624 INFO:tasks.rados.rados.0.smithi195.stdout:164: finishing write tid 4 to smithi19530025-5 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:164: finishing write tid 5 to smithi19530025-5 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:164: oid 5 updating version 0 to 29 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:164: oid 5 version 29 is already newer than 28 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:164: oid 5 version 29 is already newer than 27 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:164: oid 5 updating version 29 to 30 2024-09-16T02:14:43.627 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 5 v 30 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-16T02:14:43.628 INFO:tasks.rados.rados.0.smithi195.stdout:164: left oid 5 (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:14:43.661 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:43.658+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:43.715 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.715 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:14:43.762 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:43.759+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:43.848 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.848 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:43.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:14:43.864 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:43.862+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:43.917 INFO:tasks.rados.rados.0.smithi195.stdout:164: done (8 left) 2024-09-16T02:14:43.917 INFO:tasks.rados.rados.0.smithi195.stdout:165: done (7 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:167: done (6 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:168: done (5 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:169: done (4 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:171: done (3 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:172: done (2 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:173: done (1 left) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:174: read oid 44 snap -1 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:174: expect (ObjNum 57 snap 5 seq_num 57) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:175: read oid 7 snap -1 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:175: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:176: read oid 5 snap -1 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:176: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:177: copy_from oid 9 from oid 18 current snap is 11 2024-09-16T02:14:43.918 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 14 current snap is 11 2024-09-16T02:14:43.919 INFO:tasks.rados.rados.0.smithi195.stdout:178: seq_num 86 ranges {163840=688128,851968=540672,1392640=229376} 2024-09-16T02:14:43.937 INFO:tasks.rados.rados.0.smithi195.stdout:178: writing smithi19530025-14 from 163840 to 851968 tid 1 2024-09-16T02:14:43.948 INFO:tasks.rados.rados.0.smithi195.stdout:178: writing smithi19530025-14 from 851968 to 1392640 tid 2 2024-09-16T02:14:43.953 INFO:tasks.rados.rados.0.smithi195.stdout:178: writing smithi19530025-14 from 1392640 to 1622016 tid 3 2024-09-16T02:14:43.959 INFO:tasks.rados.rados.0.smithi195.stdout:170: finishing rollback tid 2 to smithi19530025-31 2024-09-16T02:14:43.959 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 18 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-16T02:14:43.959 INFO:tasks.rados.rados.0.smithi195.stdout:175: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-16T02:14:43.959 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 11860 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 80402 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:170: done (5 left) 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:175: done (4 left) 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:179: delete oid 48 current snap is 11 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:176: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:14:43.960 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:43.963 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:43.959+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:43.994 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:43.995 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:44.004 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 709421 2024-09-16T02:14:44.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:14:44.015 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:44.033 INFO:tasks.rados.rados.0.smithi195.stdout:177: finishing copy_from racing read to smithi19530025-9 2024-09-16T02:14:44.033 INFO:tasks.rados.rados.0.smithi195.stdout:174: expect (ObjNum 57 snap 5 seq_num 57) 2024-09-16T02:14:44.035 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 316352 2024-09-16T02:14:44.063 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1692332 2024-09-16T02:14:44.063 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:44.064 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:44.062+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:44.087 INFO:tasks.rados.rados.0.smithi195.stdout:178: finishing write tid 1 to smithi19530025-14 2024-09-16T02:14:44.087 INFO:tasks.rados.rados.0.smithi195.stdout:177: finishing copy_from to smithi19530025-9 2024-09-16T02:14:44.087 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 9 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T02:14:44.087 INFO:tasks.rados.rados.0.smithi195.stdout:178: finishing write tid 2 to smithi19530025-14 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: finishing write tid 3 to smithi19530025-14 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:174: done (4 left) 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:176: done (3 left) 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:177: done (2 left) 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:179: done (1 left) 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:180: snap_create 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: finishing write tid 4 to smithi19530025-14 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: finishing write tid 5 to smithi19530025-14 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: oid 14 updating version 0 to 92 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: oid 14 version 92 is already newer than 91 2024-09-16T02:14:44.088 INFO:tasks.rados.rados.0.smithi195.stdout:178: oid 14 updating version 92 to 93 2024-09-16T02:14:44.089 INFO:tasks.rados.rados.0.smithi195.stdout:178: oid 14 version 93 is already newer than 90 2024-09-16T02:14:44.089 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 14 v 93 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-16T02:14:44.089 INFO:tasks.rados.rados.0.smithi195.stdout:178: left oid 14 (ObjNum 86 snap 11 seq_num 86) 2024-09-16T02:14:44.142 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:44.142 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:44.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:14:44.164 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:44.158+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:44.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:44.265 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:44.259+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:44.322 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:44.442 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:44.444 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:44.446 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:44.447 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:44.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:14:44.448 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:44.442+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:44.448 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:44.449 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:44.467 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:44.468 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:44.494 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:44.494 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:44.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:44.545 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:44.543+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:44.550 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:44.550 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:44.582 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:44.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:44.612 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:44.613 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:44.625 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:44.625 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.304823103, 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051833920000000002, 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318273+0000", 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4999999999999998e-07 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318481+0000", 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020832199999999999 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318589+0000", 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107645 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318612+0000", 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3468999999999998e-05 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318803+0000", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019100200000000001 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318832+0000", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8812000000000001e-05 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323396+0000", 2024-09-16T02:14:44.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045639950000000004 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323455+0000", 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9697000000000001e-05 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 44/32 ECSubWrite(tid=29, reqid=osd.5.0:29, at_version=44'20, trim_to=0'0, roll_forward_to=43'18))", 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.334780+0000", 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.28831475, 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002947776, 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.632 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334780+0000", 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334780+0000", 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334781+0000", 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5700000000000002e-07 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334840+0000", 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8493999999999999e-05 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334939+0000", 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9512999999999994e-05 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335001+0000", 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1665e-05 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335219+0000", 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021805500000000001 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335239+0000", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0526e-05 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.337636+0000", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023965789999999998 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.337728+0000", 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2187000000000004e-05 2024-09-16T02:14:44.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.347049+0000", 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.276046486, 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030181600000000002, 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347049+0000", 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347049+0000", 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347051+0000", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9759999999999998e-06 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347078+0000", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7389000000000002e-05 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347199+0000", 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012072 2024-09-16T02:14:44.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347222+0000", 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2844999999999999e-05 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347673+0000", 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045136800000000001 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347698+0000", 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4875000000000002e-05 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349980+0000", 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002282184 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350067+0000", 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6803000000000002e-05 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 45/32 ECSubWrite(tid=31, reqid=osd.5.0:31, at_version=45'22, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.342106+0000", 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.280988867, 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032716949999999998, 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342106+0000", 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342106+0000", 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342107+0000", 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0299999999999999e-07 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342161+0000", 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3619000000000003e-05 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342311+0000", 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015091400000000001 2024-09-16T02:14:44.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342338+0000", 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7002999999999999e-05 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342741+0000", 2024-09-16T02:14:44.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040245599999999998 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.342765+0000", 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3791999999999999e-05 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.345308+0000", 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025434530000000002 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.345378+0000", 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9855000000000001e-05 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.348769+0000", 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.274326279, 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031734150000000002, 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.647 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.348769+0000", 2024-09-16T02:14:44.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.348769+0000", 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.348769+0000", 2024-09-16T02:14:44.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4300000000000003e-07 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.348829+0000", 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9327999999999998e-05 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.348985+0000", 2024-09-16T02:14:44.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015614900000000001 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.349007+0000", 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1719e-05 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.349166+0000", 2024-09-16T02:14:44.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015904800000000001 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.349222+0000", 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5951e-05 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.351846+0000", 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002623977 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.351942+0000", 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6700000000000006e-05 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.653 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.358025+0000", 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.265070379000001, 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026386230000000001, 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.654 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358025+0000", 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358025+0000", 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358025+0000", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7299999999999995e-07 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358075+0000", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9811000000000001e-05 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358206+0000", 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013082299999999999 2024-09-16T02:14:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358230+0000", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358393+0000", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016264099999999999 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358418+0000", 2024-09-16T02:14:44.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4893e-05 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360550+0000", 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021318029999999998 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360663+0000", 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011372 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.0s3 45/32 ECSubWrite(tid=12, reqid=client.4598.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=45'3))", 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.358885+0000", 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.264210654999999, 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004473543, 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358885+0000", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358885+0000", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358892+0000", 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9359999999999999e-06 2024-09-16T02:14:44.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.358917+0000", 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4479000000000001e-05 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359000+0000", 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2738000000000006e-05 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359015+0000", 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5449000000000001e-05 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359168+0000", 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015327799999999999 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359192+0000", 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3462e-05 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363233+0000", 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040412549999999997 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363358+0000", 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012494599999999999 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:44.663 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.360557+0000", 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.26253782, 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047813339999999999, 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360557+0000", 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360557+0000", 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360559+0000", 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7570000000000001e-06 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360587+0000", 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8048999999999999e-05 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360684+0000", 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6494000000000001e-05 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360707+0000", 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3068999999999999e-05 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360912+0000", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020524900000000001 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360937+0000", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4916e-05 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365250+0000", 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043134200000000001 2024-09-16T02:14:44.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365339+0000", 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8380000000000004e-05 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.360779+0000", 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.262316052999999, 2024-09-16T02:14:44.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045735089999999999, 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360779+0000", 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360779+0000", 2024-09-16T02:14:44.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360781+0000", 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360866+0000", 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5193999999999998e-05 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360977+0000", 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000110946 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.360999+0000", 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2325999999999999e-05 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361150+0000", 2024-09-16T02:14:44.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015087799999999999 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361175+0000", 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5075999999999998e-05 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365264+0000", 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040883940000000004 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365353+0000", 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9105999999999997e-05 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 45/32 ECSubWrite(tid=17, reqid=osd.8.0:17, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.260125888999999, 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056123190000000002, 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.673 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362970+0000", 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.44e-07 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363028+0000", 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7784000000000001e-05 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364099+0000", 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00107157 2024-09-16T02:14:44.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364121+0000", 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2206000000000001e-05 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364291+0000", 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017008400000000001 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364317+0000", 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5596e-05 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368520+0000", 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042029670000000002 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368582+0000", 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1568000000000004e-05 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=29, reqid=osd.7.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=45'10))", 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.259966464, 2024-09-16T02:14:44.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053661100000000003, 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363130+0000", 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5600000000000004e-07 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363151+0000", 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1073e-05 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364189+0000", 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010384350000000001 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364205+0000", 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5732000000000001e-05 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364314+0000", 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010873400000000001 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364328+0000", 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4525000000000001e-05 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368397+0000", 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040685349999999999 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368495+0000", 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8120000000000002e-05 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s2 51/32 ECSubWrite(tid=23, reqid=osd.15.0:23, at_version=51'14, trim_to=0'0, roll_forward_to=49'12))", 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.404397+0000", 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2186981039999996, 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035388160000000002, 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404397+0000", 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404397+0000", 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404398+0000", 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7300000000000005e-07 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404475+0000", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6711000000000005e-05 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404606+0000", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013092600000000001 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404629+0000", 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3952999999999999e-05 2024-09-16T02:14:44.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404795+0000", 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016517800000000001 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.404821+0000", 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.614e-05 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.407838+0000", 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030173610000000001 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.407936+0000", 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7774000000000003e-05 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.688 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s2 51/32 ECSubWrite(tid=24, reqid=osd.15.0:23, at_version=0'0, trim_to=0'0, roll_forward_to=51'14))", 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.418389+0000", 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2047057499999996, 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025264020000000001, 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418389+0000", 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418389+0000", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418392+0000", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2129999999999998e-06 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418423+0000", 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0849000000000002e-05 2024-09-16T02:14:44.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418511+0000", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8064000000000003e-05 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418533+0000", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2476e-05 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418738+0000", 2024-09-16T02:14:44.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020488400000000001 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.418762+0000", 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4439000000000001e-05 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.420820+0000", 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002057447 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.420916+0000", 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.603e-05 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.693 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s2 52/32 ECSubWrite(tid=25, reqid=client.4598.0:397, at_version=52'15, trim_to=0'0, roll_forward_to=51'14))", 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6977291770000003, 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048112190000000003, 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925367+0000", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3600000000000002e-07 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925676+0000", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030898500000000001 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925831+0000", 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015518199999999999 2024-09-16T02:14:44.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925908+0000", 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6587000000000004e-05 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.926247+0000", 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033902999999999998 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.926275+0000", 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7926999999999999e-05 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.930095+0000", 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038202650000000002 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.930177+0000", 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2406999999999999e-05 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=22, reqid=client.4598.0:404, at_version=52'7, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6554656780000001, 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013916968, 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967631+0000", 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967931+0000", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030039800000000001 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979030+0000", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011099121999999999 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979051+0000", 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0577e-05 2024-09-16T02:14:44.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979192+0000", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140526 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979207+0000", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981459+0000", 2024-09-16T02:14:44.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022516110000000002 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981546+0000", 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7508000000000003e-05 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=23, reqid=client.4598.0:405, at_version=52'8, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6444728820000001, 2024-09-16T02:14:44.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050943480000000003, 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:14:44.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978624+0000", 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978992+0000", 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036885999999999999 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982073+0000", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030808860000000001 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982089+0000", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5310000000000001e-05 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982188+0000", 2024-09-16T02:14:44.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9246999999999994e-05 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982203+0000", 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4600000000000001e-05 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.983631+0000", 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014287810000000001 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.983717+0000", 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5371000000000006e-05 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=24, reqid=client.4598.0:406, at_version=52'9, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.981992+0000", 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6411034199999999, 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003908145, 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.708 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981992+0000", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981992+0000", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981993+0000", 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982040+0000", 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6495000000000001e-05 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984285+0000", 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022452879999999998 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984300+0000", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5291999999999999e-05 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984406+0000", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010623700000000001 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984424+0000", 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7142999999999998e-05 2024-09-16T02:14:44.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.985813+0000", 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001389363 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.985900+0000", 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7007000000000002e-05 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=25, reqid=client.4598.0:407, at_version=52'10, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6388826920000001, 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34897630299999999, 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984214+0000", 2024-09-16T02:14:44.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.384e-06 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984244+0000", 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0549000000000002e-05 2024-09-16T02:14:44.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331741+0000", 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34749696499999999 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331756+0000", 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4873e-05 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331921+0000", 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016467900000000001 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331952+0000", 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0895000000000002e-05 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333108+0000", 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001156633 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333189+0000", 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0325000000000001e-05 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 52/32 ECSubWrite(tid=226, reqid=client.4598.0:412, at_version=52'47, trim_to=0'0, roll_forward_to=51'45))", 2024-09-16T02:14:44.718 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.993999+0000", 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6290957719999999, 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043509730000000002, 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.993999+0000", 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.993999+0000", 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.994001+0000", 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3069999999999999e-06 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.994837+0000", 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00083666199999999995 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.994991+0000", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154022 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.995015+0000", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3986e-05 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.995191+0000", 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017578200000000001 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.995214+0000", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2793e-05 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.998257+0000", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030430169999999999 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.998350+0000", 2024-09-16T02:14:44.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3404000000000002e-05 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 52/32 ECSubWrite(tid=227, reqid=client.4598.0:413, at_version=52'48, trim_to=0'0, roll_forward_to=51'45))", 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.999417+0000", 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6236779170000002, 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024563279999999998, 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:44.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999417+0000", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999417+0000", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999419+0000", 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.874e-06 2024-09-16T02:14:44.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999611+0000", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019180999999999999 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999701+0000", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9730999999999999e-05 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999717+0000", 2024-09-16T02:14:44.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6643e-05 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999875+0000", 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000157405 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.999890+0000", 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5463000000000001e-05 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:44.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.001782+0000", 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018921319999999999 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.001874+0000", 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1269999999999999e-05 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:44.727 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:44.728 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:44.729 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:44.730 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:44.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:14:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:44.731 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:44.731 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:44.723+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:44.745 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:44.745 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:44.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:44.781 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:44.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:44.810+0000 7f8013fff640 0 -- 172.21.15.3:0/1705897908 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f80142274d0 msgr2=0x7f80142298c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:44.827 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:44.830 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:44.822+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:44.858 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:44.858 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:44.858 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:44.858 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:44.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:14:44.878 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:44.878 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:178: done (1 left) 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:180: done (0 left) 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:181: rmattr oid 12 current snap is 12 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:181: done (0 left) 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 13 current snap is 12 2024-09-16T02:14:44.925 INFO:tasks.rados.rados.0.smithi195.stdout:182: seq_num 88 ranges {1376256=720896,2097152=442368,2539520=327680} 2024-09-16T02:14:44.930 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:44.927+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:44.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:14:44.958 INFO:tasks.rados.rados.0.smithi195.stdout:182: writing smithi19530025-13 from 1376256 to 2097152 tid 1 2024-09-16T02:14:44.968 INFO:tasks.rados.rados.0.smithi195.stdout:182: writing smithi19530025-13 from 2097152 to 2539520 tid 2 2024-09-16T02:14:44.974 INFO:tasks.rados.rados.0.smithi195.stdout:182: writing smithi19530025-13 from 2539520 to 2867200 tid 3 2024-09-16T02:14:44.974 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 7 current snap is 12 2024-09-16T02:14:44.974 INFO:tasks.rados.rados.0.smithi195.stdout:183: seq_num 89 ranges {114688=737280,851968=573440,1425408=196608} 2024-09-16T02:14:44.985 INFO:tasks.rados.rados.0.smithi195.stdout:183: writing smithi19530025-7 from 114688 to 851968 tid 1 2024-09-16T02:14:44.992 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:44.993 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:44.993 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:44.993 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:44.993 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:44.993 INFO:tasks.rados.rados.0.smithi195.stdout:183: writing smithi19530025-7 from 851968 to 1425408 tid 2 2024-09-16T02:14:44.995 INFO:tasks.rados.rados.0.smithi195.stdout:183: writing smithi19530025-7 from 1425408 to 1622016 tid 3 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 28 current snap is 12 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: finishing write tid 1 to smithi19530025-13 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: finishing write tid 2 to smithi19530025-13 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: finishing write tid 3 to smithi19530025-13 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: finishing write tid 4 to smithi19530025-13 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: finishing write tid 5 to smithi19530025-13 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: oid 13 updating version 0 to 100 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: oid 13 version 100 is already newer than 99 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: oid 13 version 100 is already newer than 97 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: oid 13 version 100 is already newer than 98 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 13 v 100 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-16T02:14:44.996 INFO:tasks.rados.rados.0.smithi195.stdout:182: left oid 13 (ObjNum 88 snap 12 seq_num 88) 2024-09-16T02:14:44.997 INFO:tasks.rados.rados.0.smithi195.stdout:184: seq_num 90 ranges {0=753664,753664=720896,1474560=49152} 2024-09-16T02:14:45.001 INFO:tasks.rados.rados.0.smithi195.stdout:184: writing smithi19530025-28 from 0 to 753664 tid 1 2024-09-16T02:14:45.005 INFO:tasks.rados.rados.0.smithi195.stdout:184: writing smithi19530025-28 from 753664 to 1474560 tid 2 2024-09-16T02:14:45.006 INFO:tasks.rados.rados.0.smithi195.stdout:184: writing smithi19530025-28 from 1474560 to 1523712 tid 3 2024-09-16T02:14:45.006 INFO:tasks.rados.rados.0.smithi195.stdout:182: done (2 left) 2024-09-16T02:14:45.006 INFO:tasks.rados.rados.0.smithi195.stdout:185: snap_remove snap 10 2024-09-16T02:14:45.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:14:45.032 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:45.027+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:45.133 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:45.127+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.136 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4293.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.965670537000001, 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084256020000000008, 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4293", 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168346+0000", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168352+0000", 2024-09-16T02:14:45.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0299999999999999e-06 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168366+0000", 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168399+0000", 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2588e-05 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168427+0000", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8371e-05 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168562+0000", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134337 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.168584+0000", 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2011999999999999e-05 2024-09-16T02:14:45.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173264+0000", 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046804209999999997 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173570+0000", 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000305638 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2", 2024-09-16T02:14:45.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.173890+0000", 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000320122 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176184+0000", 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022945679999999999 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176505+0000", 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032044099999999998 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176650+0000", 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145253 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:08.176771+0000", 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012114500000000001 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:192 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~557056] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.819206830999999, 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015517632999999999, 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.144 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 192 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:14:45.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314812+0000", 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029999999999999e-06 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315788+0000", 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097551599999999995 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315825+0000", 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6742000000000002e-05 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315847+0000", 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2396e-05 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316018+0000", 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170729 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316179+0000", 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016178000000000001 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318923+0000", 2024-09-16T02:14:45.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002743381 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071732469999999998 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330202+0000", 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004106391 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330327+0000", 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012444800000000001 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:193 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.812604253, 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094258069999999996, 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.150 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321415+0000", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8789999999999999e-06 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321889+0000", 2024-09-16T02:14:45.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047466400000000002 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321924+0000", 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4134999999999998e-05 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321944+0000", 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0060000000000001e-05 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322086+0000", 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014260999999999999 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322329+0000", 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024320400000000001 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324137+0000", 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018075879999999999 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328050+0000", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039127119999999996 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330645+0000", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025952829999999999 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330838+0000", 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019267199999999999 2024-09-16T02:14:45.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:194 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (56) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.812027807, 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011982552, 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.156 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 194 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321990+0000", 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.756e-06 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322010+0000", 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9443e-05 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322033+0000", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322049+0000", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6201999999999998e-05 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326191+0000", 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041419430000000004 2024-09-16T02:14:45.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326397+0000", 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020605800000000001 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327141+0000", 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074450000000000004 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328976+0000", 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018343109999999999 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333836+0000", 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048599389999999998 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333971+0000", 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013532799999999999 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:195 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.811933321000001, 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014685304, 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 195 2024-09-16T02:14:45.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322084+0000", 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322092+0000", 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6529999999999994e-06 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322111+0000", 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8909000000000001e-05 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322124+0000", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3589e-05 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327750+0000", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056258819999999996 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327883+0000", 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013255499999999999 2024-09-16T02:14:45.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334789+0000", 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069061050000000001 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335013+0000", 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022373400000000001 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.336768+0000", 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001755454 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.784358379, 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045915540000000003, 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.167 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349658+0000", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.349659+0000", 2024-09-16T02:14:45.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0899999999999999e-06 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350269+0000", 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061005800000000002 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350367+0000", 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8338999999999995e-05 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350395+0000", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7313000000000001e-05 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350521+0000", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126945 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350539+0000", 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7419e-05 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354142+0000", 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036026909999999999 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354249+0000", 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107699 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:242 3.8s0 3:165ce5f6:::smithi19530025-40:head [copy-from ver 0] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.799058751, 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011828727000000001, 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334961+0000", 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.326e-06 2024-09-16T02:14:45.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334977+0000", 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6384000000000001e-05 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335012+0000", 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4465000000000003e-05 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335033+0000", 2024-09-16T02:14:45.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1305999999999999e-05 2024-09-16T02:14:45.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336134+0000", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011016540000000001 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336397+0000", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026217800000000001 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.346786+0000", 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010389414 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.177 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:243 3.8s0 3:165ce5f6:::smithi19530025-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.798931298999999, 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01238621, 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335086+0000", 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.66e-06 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335094+0000", 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9829999999999996e-06 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335118+0000", 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3716000000000001e-05 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335135+0000", 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7102999999999999e-05 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336819+0000", 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001684051 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336943+0000", 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012403199999999999 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.346988+0000", 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010044599 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347205+0000", 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021740900000000001 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347471+0000", 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026565699999999998 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 44/32 ECSubWrite(tid=30, reqid=osd.3.0:30, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.348092+0000", 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.785923874, 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058783630000000002, 2024-09-16T02:14:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348092+0000", 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348092+0000", 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348093+0000", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6699999999999999e-07 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348145+0000", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2701000000000002e-05 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348283+0000", 2024-09-16T02:14:45.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013759899999999999 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348307+0000", 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3811999999999998e-05 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348598+0000", 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291487 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348623+0000", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4479000000000001e-05 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.353878+0000", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052547849999999997 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.353971+0000", 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3033000000000002e-05 2024-09-16T02:14:45.188 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 44/32 ECSubWrite(tid=31, reqid=osd.3.0:31, at_version=44'28, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.348150+0000", 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.785865703000001, 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071767910000000001, 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348150+0000", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348150+0000", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348151+0000", 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8299999999999997e-07 2024-09-16T02:14:45.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348170+0000", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8723999999999999e-05 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.351968+0000", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037986859999999999 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.351994+0000", 2024-09-16T02:14:45.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6038999999999999e-05 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.352168+0000", 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017330700000000001 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.352196+0000", 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8622000000000001e-05 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.355192+0000", 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029959079999999998 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.355327+0000", 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135022 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 44/32 ECSubWrite(tid=42, reqid=osd.11.0:42, at_version=44'33, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:14:45.193 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.349492+0000", 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.784523932999999, 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044421069999999998, 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349492+0000", 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349492+0000", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349493+0000", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3600000000000004e-07 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349547+0000", 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4168e-05 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349633+0000", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5693999999999996e-05 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349654+0000", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1809000000000001e-05 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349851+0000", 2024-09-16T02:14:45.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019652999999999999 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349876+0000", 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5245999999999999e-05 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.353884+0000", 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040079019999999998 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.353934+0000", 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0222000000000001e-05 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 44/32 ECSubWrite(tid=43, reqid=osd.11.0:43, at_version=44'35, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.350568+0000", 2024-09-16T02:14:45.198 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.783448128, 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044536130000000004, 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350568+0000", 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350568+0000", 2024-09-16T02:14:45.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350570+0000", 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.55e-06 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350612+0000", 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2552000000000002e-05 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350681+0000", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9210999999999999e-05 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.350702+0000", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1032000000000001e-05 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.351903+0000", 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001200951 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.351929+0000", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5738999999999999e-05 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.354944+0000", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030151140000000002 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.355022+0000", 2024-09-16T02:14:45.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7464000000000002e-05 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 44/32 ECSubWrite(tid=46, reqid=client.4598.0:245, at_version=44'38, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.430400+0000", 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.703616422, 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039093549999999998, 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430400+0000", 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430400+0000", 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430402+0000", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8560000000000001e-06 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430794+0000", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039233300000000003 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430869+0000", 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4794000000000003e-05 2024-09-16T02:14:45.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.430883+0000", 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4718999999999999e-05 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431044+0000", 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016073199999999999 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431069+0000", 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5349000000000001e-05 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434207+0000", 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031378619999999999 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434309+0000", 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010171 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s5 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.359143+0000", 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.774872818, 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053593160000000003, 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359143+0000", 2024-09-16T02:14:45.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359143+0000", 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359144+0000", 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359182+0000", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7614999999999998e-05 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359337+0000", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015494099999999999 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359359+0000", 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2206000000000001e-05 2024-09-16T02:14:45.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359559+0000", 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019983300000000001 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.359639+0000", 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0723999999999995e-05 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364412+0000", 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047726829999999998 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364503+0000", 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0643999999999995e-05 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:45.212 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361557+0000", 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.772459435, 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071405399999999999, 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361557+0000", 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361557+0000", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361557+0000", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8699999999999995e-07 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361604+0000", 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6394999999999999e-05 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361734+0000", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012997599999999999 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361782+0000", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8494e-05 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362006+0000", 2024-09-16T02:14:45.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224253 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362033+0000", 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6693e-05 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368618+0000", 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065850179999999998 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9123999999999997e-05 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s5 45/32 ECSubWrite(tid=34, reqid=osd.3.0:34, at_version=45'32, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.365279+0000", 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.768737392, 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048021940000000001, 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365279+0000", 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365279+0000", 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365280+0000", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.654e-06 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365377+0000", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6262999999999996e-05 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.366361+0000", 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00098470999999999997 2024-09-16T02:14:45.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.366383+0000", 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2022e-05 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367021+0000", 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00063770199999999997 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367048+0000", 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6812e-05 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369987+0000", 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029389030000000001 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.370081+0000", 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4128000000000004e-05 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.221 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s5 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.365402+0000", 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.768614191999999, 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047280639999999997, 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365402+0000", 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365402+0000", 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365403+0000", 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6299999999999993e-07 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365441+0000", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7848000000000001e-05 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.366826+0000", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001385196 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.366854+0000", 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7793e-05 2024-09-16T02:14:45.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367018+0000", 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016407800000000001 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367043+0000", 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4779000000000001e-05 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.370041+0000", 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029981669999999999 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.370130+0000", 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9240000000000006e-05 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:45.226 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.365635+0000", 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.768381134, 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043674940000000004, 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365635+0000", 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365635+0000", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365637+0000", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7439999999999999e-06 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365690+0000", 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2994000000000001e-05 2024-09-16T02:14:45.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365789+0000", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9199999999999999e-05 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365813+0000", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4450000000000001e-05 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365974+0000", 2024-09-16T02:14:45.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016101000000000001 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.366000+0000", 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5168999999999999e-05 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369899+0000", 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038995710000000001 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.370002+0000", 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103356 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:257 3.8s0 3:177f2fba:::smithi19530025-45:head [delete] snapc 6=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e45)", 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.389744+0000", 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.744272512, 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043995470000000002, 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.231 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 257 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389744+0000", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389744+0000", 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389746+0000", 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2060000000000001e-06 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389758+0000", 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.186e-05 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389787+0000", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9624e-05 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389802+0000", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4677e-05 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.389922+0000", 2024-09-16T02:14:45.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119806 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.390082+0000", 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016025100000000001 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.390835+0000", 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075319700000000005 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.392442+0000", 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001606711 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.393965+0000", 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001522818 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.394143+0000", 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017839700000000001 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.236 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.15.21:1 3.8s0 3:1cf09812:::smithi19530025-15:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2203288260000003, 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088887500000000008, 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913691+0000", 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5030000000000002e-06 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913715+0000", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4232000000000001e-05 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913805+0000", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0000999999999994e-05 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913831+0000", 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843e-05 2024-09-16T02:14:45.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.914184+0000", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035352300000000001 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.914197+0000", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2326e-05 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.919183+0000", 2024-09-16T02:14:45.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049860640000000001 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.919371+0000", 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000188178 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.922576+0000", 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032050799999999999 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.242 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.242 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:45.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:14:45.244 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:45.238+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:45.344 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:45.342+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:45.370 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:45.370 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:45.370 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:45.371 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:45.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:14:45.445 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:45.443+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:45.543 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:45.543 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:45.543 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:45.544 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:45.544 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:45.546 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:45.546+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:45.557 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:14:45.646 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:45.646+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.725 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.443937506000001, 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.726 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:14:45.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:14:45.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.439715060000001, 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:14:45.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:14:45.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.439419750999999, 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006345626, 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.735 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283859+0000", 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283921+0000", 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1947999999999997e-05 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284076+0000", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015468599999999999 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284095+0000", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8686e-05 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285435+0000", 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001340639 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285464+0000", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8411999999999998e-05 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290127+0000", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046638529999999999 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290204+0000", 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079000000000001e-05 2024-09-16T02:14:45.739 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=1, reqid=client.4598.0:17, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.437386143000001, 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048188320000000003, 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.740 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285894+0000", 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.667e-06 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285942+0000", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8133999999999997e-05 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286032+0000", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9883000000000007e-05 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286080+0000", 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7605000000000002e-05 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286237+0000", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015714200000000001 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286262+0000", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5012e-05 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:14:45.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043960889999999997 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290710+0000", 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2299999999999997e-05 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:45.745 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.426963128000001, 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:45.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:14:45.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.424298747999998, 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.751 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:14:45.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.756 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.424239528000001, 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:14:45.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.761 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.423832223000002, 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:14:45.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.421951478, 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:14:45.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.421675914000001, 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:45.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:14:45.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:14:45.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.778 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.420350516999999, 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00583893, 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:45.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302929+0000", 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.717e-06 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302970+0000", 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1564999999999999e-05 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303048+0000", 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8059000000000006e-05 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303070+0000", 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282999999999999e-05 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306196+0000", 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00312663 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306213+0000", 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6747000000000001e-05 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308730+0000", 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002516796 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308766+0000", 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6133000000000001e-05 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=6, reqid=client.4598.0:20, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.418871763999999, 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045402039999999999, 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304406+0000", 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304408+0000", 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3599999999999999e-06 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304438+0000", 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9399000000000001e-05 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304525+0000", 2024-09-16T02:14:45.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6935999999999996e-05 2024-09-16T02:14:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304548+0000", 2024-09-16T02:14:45.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3272e-05 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304730+0000", 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018221900000000001 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.304756+0000", 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5908999999999999e-05 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308884+0000", 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041276189999999999 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308946+0000", 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2490000000000006e-05 2024-09-16T02:14:45.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.373144266000001, 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046039000000000002, 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350134+0000", 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:14:45.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350792+0000", 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065815600000000002 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350844+0000", 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1777999999999998e-05 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350861+0000", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6826000000000001e-05 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351025+0000", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164623 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351055+0000", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9672999999999999e-05 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354677+0000", 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036216830000000001 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354737+0000", 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0534e-05 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 44/32 ECSubWrite(tid=31, reqid=osd.3.0:31, at_version=44'28, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.347885+0000", 2024-09-16T02:14:45.795 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.375392653, 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042402230000000004, 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347885+0000", 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347885+0000", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347887+0000", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.77e-06 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347937+0000", 2024-09-16T02:14:45.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9960999999999998e-05 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348026+0000", 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9530000000000005e-05 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.348051+0000", 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4465999999999999e-05 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349483+0000", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014324489999999999 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.349520+0000", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6981999999999999e-05 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.352030+0000", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002509875 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.352125+0000", 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5190000000000002e-05 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=45, reqid=client.4598.0:244, at_version=44'36, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.303812684, 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013820059000000001, 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.800 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419467+0000", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9400000000000001e-06 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419693+0000", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000226312 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431387+0000", 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011694154999999999 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431404+0000", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6945000000000001e-05 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431560+0000", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015586199999999999 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431575+0000", 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4979e-05 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433182+0000", 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016072700000000001 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433285+0000", 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010259600000000001 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=46, reqid=client.4598.0:245, at_version=44'38, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.292195570000001, 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060040980000000002, 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431084+0000", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431346+0000", 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026225100000000002 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434399+0000", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030528320000000001 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434422+0000", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3326e-05 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434546+0000", 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411200000000001 2024-09-16T02:14:45.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434562+0000", 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5075999999999999e-05 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437029+0000", 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024676699999999999 2024-09-16T02:14:45.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437086+0000", 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7008e-05 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=48, reqid=client.4598.0:247, at_version=44'40, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.288256476000001, 2024-09-16T02:14:45.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29780827999999998, 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:14:45.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435023+0000", 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435063+0000", 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731424+0000", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29636148400000001 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731439+0000", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.453e-05 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731595+0000", 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015660699999999999 2024-09-16T02:14:45.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731614+0000", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8243000000000001e-05 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732758+0000", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011446360000000001 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732829+0000", 2024-09-16T02:14:45.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1191000000000003e-05 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.366383608, 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080490270000000003, 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356895+0000", 2024-09-16T02:14:45.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356948+0000", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3032000000000003e-05 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357075+0000", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012766899999999999 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357100+0000", 2024-09-16T02:14:45.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5086e-05 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357282+0000", 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018186299999999999 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7348999999999999e-05 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364849+0000", 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075397019999999997 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364943+0000", 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3709999999999996e-05 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:45.820 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.361632215, 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070518059999999999, 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361646+0000", 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361693+0000", 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6882000000000003e-05 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361805+0000", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112046 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361834+0000", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9215999999999999e-05 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362268+0000", 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043351899999999999 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362294+0000", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6142000000000001e-05 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368607+0000", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063134869999999996 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:14:45.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9844000000000002e-05 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:14:45.825 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.359557299, 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059443869999999998, 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:14:45.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363722+0000", 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.733e-06 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363805+0000", 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2776999999999997e-05 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:45.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363906+0000", 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010156 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363931+0000", 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4966000000000001e-05 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365907+0000", 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019751790000000001 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365933+0000", 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205999999999999e-05 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369569+0000", 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036364890000000001 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:45.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369665+0000", 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5476999999999994e-05 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:45.831 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:45.832 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:45.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:14:45.833 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:45.830+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:45.921 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:14:45.930 INFO:tasks.rados.rados.0.smithi195.stdout:185: done (2 left) 2024-09-16T02:14:45.930 INFO:tasks.rados.rados.0.smithi195.stdout:186: setattr oid 23 current snap is 12 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: finishing write tid 1 to smithi19530025-7 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: finishing write tid 2 to smithi19530025-7 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: finishing write tid 3 to smithi19530025-7 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: finishing write tid 4 to smithi19530025-7 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: finishing write tid 5 to smithi19530025-7 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: oid 7 updating version 0 to 12 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: oid 7 version 12 is already newer than 10 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: oid 7 updating version 12 to 13 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: oid 7 version 13 is already newer than 11 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 13 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:183: left oid 7 (ObjNum 89 snap 12 seq_num 89) 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:184: finishing write tid 1 to smithi19530025-28 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:184: finishing write tid 2 to smithi19530025-28 2024-09-16T02:14:45.931 INFO:tasks.rados.rados.0.smithi195.stdout:184: finishing write tid 3 to smithi19530025-28 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: finishing write tid 4 to smithi19530025-28 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: finishing write tid 5 to smithi19530025-28 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: oid 28 updating version 0 to 15 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: oid 28 version 15 is already newer than 14 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: oid 28 updating version 15 to 17 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: oid 28 version 17 is already newer than 16 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 28 v 17 (ObjNum 90 snap 12 seq_num 90) dirty exists 2024-09-16T02:14:45.932 INFO:tasks.rados.rados.0.smithi195.stdout:184: left oid 28 (ObjNum 90 snap 12 seq_num 90) 2024-09-16T02:14:45.933 INFO:tasks.rados.rados.0.smithi195.stdout:183: done (2 left) 2024-09-16T02:14:45.933 INFO:tasks.rados.rados.0.smithi195.stdout:184: done (1 left) 2024-09-16T02:14:45.933 INFO:tasks.rados.rados.0.smithi195.stdout:187: read oid 33 snap -1 2024-09-16T02:14:45.933 INFO:tasks.rados.rados.0.smithi195.stdout:187: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T02:14:45.933 INFO:tasks.rados.rados.0.smithi195.stdout:188: snap_create 2024-09-16T02:14:45.933 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:45.931+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:45.944 INFO:tasks.rados.rados.0.smithi195.stdout:187: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T02:14:45.951 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 395661 2024-09-16T02:14:45.951 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:45.957 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:45.963 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 23 v 55 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-16T02:14:46.007 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:46.007 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:46.007 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:46.007 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:46.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:14:46.034 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:46.031+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:46.135 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:46.131+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:46.143 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:46.143 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:46.143 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:46.143 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:46.143 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:46.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:14:46.236 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:46.230+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:46.283 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:14:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:14:46.283 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 40.209182255000002, 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:14:46.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:46.290 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.704412187999999, 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020204412000000001, 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576033+0000", 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6599999999999995e-07 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576055+0000", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1931999999999999e-05 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576066+0000", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1369000000000001e-05 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576074+0000", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5630000000000003e-06 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594265+0000", 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018191161000000001 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594479+0000", 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021409700000000001 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594685+0000", 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000205708 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595917+0000", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232564 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596058+0000", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014070900000000001 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596135+0000", 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6878000000000005e-05 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596237+0000", 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101665 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.483105657999999, 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019987173, 2024-09-16T02:14:46.296 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797340+0000", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797353+0000", 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3156000000000001e-05 2024-09-16T02:14:46.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797372+0000", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8382000000000001e-05 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797386+0000", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4262e-05 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811114+0000", 2024-09-16T02:14:46.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013728557000000001 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811354+0000", 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023977499999999999 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811632+0000", 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027806899999999998 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815429+0000", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037971509999999999 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817182+0000", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001752382 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817262+0000", 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0240000000000004e-05 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817326+0000", 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4022000000000005e-05 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.483034175999997, 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020141841000000001, 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.302 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797412+0000", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.243e-06 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797542+0000", 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077400000000001 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797562+0000", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9644e-05 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797577+0000", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812322+0000", 2024-09-16T02:14:46.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014744597999999999 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812456+0000", 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134466 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812671+0000", 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000214537 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815454+0000", 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027836179999999999 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817415+0000", 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019603979999999999 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817490+0000", 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5012000000000006e-05 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817552+0000", 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2331999999999999e-05 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.482841471, 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020132852, 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.308 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:14:46.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797605+0000", 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7069999999999999e-06 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797620+0000", 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5585000000000001e-05 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797639+0000", 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8212000000000001e-05 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797658+0000", 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9514999999999999e-05 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813882+0000", 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016224045999999999 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814108+0000", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022626600000000001 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814462+0000", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035324400000000001 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815978+0000", 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015162000000000001 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817614+0000", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001636366 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817684+0000", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9701000000000002e-05 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817736+0000", 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2009999999999998e-05 2024-09-16T02:14:46.313 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.371367290000002, 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019924002999999999, 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 122 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909078+0000", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.23e-07 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909084+0000", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000002e-06 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909094+0000", 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8330000000000006e-06 2024-09-16T02:14:46.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909104+0000", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0781999999999999e-05 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922447+0000", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013342309 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922571+0000", 2024-09-16T02:14:46.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012409699999999999 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922765+0000", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000193869 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926685+0000", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003919952 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928862+0000", 2024-09-16T02:14:46.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002177216 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928940+0000", 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8281000000000005e-05 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.929001+0000", 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1030999999999999e-05 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.371326449999998, 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.320 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:14:46.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:14:46.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:14:46.325 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 39.371202056000001, 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:14:46.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:14:46.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:1 3.3s0 3:c481b30a:::smithi19530025-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.003049419, 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021556188, 2024-09-16T02:14:46.331 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277399+0000", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4400000000000001e-06 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277527+0000", 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012811100000000001 2024-09-16T02:14:46.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277557+0000", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0645000000000002e-05 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277594+0000", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6848999999999999e-05 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277805+0000", 2024-09-16T02:14:46.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021101299999999999 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.278199+0000", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039380300000000002 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.280065+0000", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018661680000000001 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286512+0000", 2024-09-16T02:14:46.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064472219999999998 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298756+0000", 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012244026 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298951+0000", 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194911 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:2 3.3s0 3:c481b30a:::smithi19530025-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.002795549999998, 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022809754000000002, 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.337 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277650+0000", 2024-09-16T02:14:46.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277657+0000", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277675+0000", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8148999999999999e-05 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277691+0000", 2024-09-16T02:14:46.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5943e-05 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282023+0000", 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004332247 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282308+0000", 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028409399999999998 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285666+0000", 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033580379999999998 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.292774+0000", 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071080379999999997 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300235+0000", 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074610759999999996 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300459+0000", 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224033 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.002630878000002, 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.342 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:14:46.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.999152485, 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:14:46.347 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:14:46.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:14:46.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.353 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.998030312000001, 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853072, 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282416+0000", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282453+0000", 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7261000000000001e-05 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298874+0000", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016420846999999999 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298893+0000", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9695e-05 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299501+0000", 2024-09-16T02:14:46.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000607509 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299522+0000", 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1039e-05 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303194+0000", 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036718020000000001 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303267+0000", 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3598999999999998e-05 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.997509476000001, 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.358 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:14:46.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:14:46.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.997413257000002, 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:14:46.364 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:14:46.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:14:46.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:58 3.2s0 3:431ee5ca:::smithi19530025-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.965383902999999, 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021031579000000002, 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.370 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 58 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:14:46.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315062+0000", 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315685+0000", 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062327200000000002 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315711+0000", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5519000000000001e-05 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315731+0000", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9775e-05 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322701+0000", 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069698679999999997 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323044+0000", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034300800000000001 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324822+0000", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017778659999999999 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.329203+0000", 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043815149999999999 2024-09-16T02:14:46.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.335988+0000", 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067852160000000002 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.336092+0000", 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103927 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.375 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:59 3.2s0 3:431ee5ca:::smithi19530025-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.964674001999999, 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022592232, 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 59 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8729999999999999e-06 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315782+0000", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2629999999999992e-06 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315822+0000", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0095000000000002e-05 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315843+0000", 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0905999999999999e-05 2024-09-16T02:14:46.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326893+0000", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011050249999999999 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.327039+0000", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145723 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328011+0000", 2024-09-16T02:14:46.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097276899999999996 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.330713+0000", 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027020989999999999 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338142+0000", 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074284329999999999 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338363+0000", 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022082100000000001 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.964574836000001, 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:14:46.381 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:14:46.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:14:46.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.912839089, 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:14:46.386 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:14:46.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:14:46.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.908551408999999, 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:14:46.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:14:46.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:14:46.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:14:46.393 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:14:46.394 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:46.395 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:14:46.396 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:46.387+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:46.496 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:46.490+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:46.529 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:46.529 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:46.530 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:46.530 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:46.544 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:14:46.597 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:46.595+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:46.666 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:46.666 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:46.667 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:46.667 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:46.667 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:46.680 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:14:46.698 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:46.691+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:46.799 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:46.794+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 38.658911316999998, 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.852 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.853 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:14:46.854 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:14:46.855 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.856 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.857 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 37.683965897, 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037540080000000001, 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.858 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 5 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.859 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165477+0000", 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165479+0000", 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.323e-06 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.860 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165493+0000", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3295999999999999e-05 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165525+0000", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2118e-05 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165542+0000", 2024-09-16T02:14:46.861 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7176e-05 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165651+0000", 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010935200000000001 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.165890+0000", 2024-09-16T02:14:46.862 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023896599999999999 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.166148+0000", 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025728900000000001 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.167513+0000", 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013649999999999999 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.863 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.168951+0000", 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014381719999999999 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.169111+0000", 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000160175 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.864 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:09.169231+0000", 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120141 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.673885079999998, 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066455200000000002, 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.865 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.866 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175561+0000", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7030000000000002e-06 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175575+0000", 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3889e-05 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.867 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175608+0000", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3704000000000001e-05 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175627+0000", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175763+0000", 2024-09-16T02:14:46.868 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013625000000000001 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175774+0000", 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.116e-05 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179478+0000", 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037039989999999999 2024-09-16T02:14:46.869 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179657+0000", 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179538 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179882+0000", 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022455700000000001 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:14:46.870 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181213+0000", 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013309039999999999 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182065+0000", 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085171900000000002 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182143+0000", 2024-09-16T02:14:46.871 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7904999999999994e-05 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182203+0000", 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0909000000000002e-05 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 38/32 ECSubWrite(tid=14, reqid=client.4598.0:197, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 19.522302717999999, 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058517350000000003, 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327142+0000", 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.59e-06 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327187+0000", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5467999999999999e-05 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327289+0000", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102086 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327312+0000", 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2849000000000001e-05 2024-09-16T02:14:46.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327454+0000", 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014216700000000001 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327467+0000", 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009e-05 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332903+0000", 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054355560000000002 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332992+0000", 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9010000000000003e-05 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:14:46.877 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.506319885, 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095323660000000004, 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:14:46.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343124+0000", 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.06e-06 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343143+0000", 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8842e-05 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343322+0000", 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179414 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343345+0000", 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2629e-05 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343569+0000", 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022398900000000001 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343596+0000", 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6489e-05 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352536+0000", 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089404250000000001 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352656+0000", 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119518 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/32 ECSubWrite(tid=19, reqid=osd.6.0:19, at_version=44'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.506230638, 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083893449999999994, 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.882 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:14:46.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343214+0000", 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343264+0000", 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0175e-05 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343378+0000", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114159 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343396+0000", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7926000000000001e-05 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343513+0000", 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117056 2024-09-16T02:14:46.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343528+0000", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4324999999999999e-05 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351501+0000", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079735710000000005 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351602+0000", 2024-09-16T02:14:46.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010059 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.504895726999999, 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082631800000000002, 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344549+0000", 2024-09-16T02:14:46.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344608+0000", 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8953999999999998e-05 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344655+0000", 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7268999999999998e-05 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344677+0000", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1382e-05 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344840+0000", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016295099999999999 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344865+0000", 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5035e-05 2024-09-16T02:14:46.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352714+0000", 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078496989999999999 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352810+0000", 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6146999999999999e-05 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.891 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=189, reqid=osd.13.0:188, at_version=0'0, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.354058+0000", 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 13.495384835999999, 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003899226, 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354058+0000", 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354058+0000", 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354060+0000", 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.793e-06 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354086+0000", 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6432e-05 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354130+0000", 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.3835000000000003e-05 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354154+0000", 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3689e-05 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354342+0000", 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018793299999999999 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354367+0000", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4766e-05 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.357829+0000", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0034624429999999999 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.357957+0000", 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012833500000000001 2024-09-16T02:14:46.896 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 45/32 ECSubWrite(tid=21, reqid=osd.6.0:21, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.344708+0000", 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.504734633, 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037874649999999998, 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344708+0000", 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344708+0000", 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344709+0000", 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2e-07 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344782+0000", 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3281000000000005e-05 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344909+0000", 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000126439 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.344928+0000", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9536000000000001e-05 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.345100+0000", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017186400000000001 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.345125+0000", 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4465999999999999e-05 2024-09-16T02:14:46.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348412+0000", 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032878260000000002 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348496+0000", 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3533000000000001e-05 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.502080381000001, 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009462359, 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.902 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347364+0000", 2024-09-16T02:14:46.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.623e-06 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347386+0000", 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1667000000000001e-05 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347450+0000", 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4116999999999996e-05 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347464+0000", 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4347e-05 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348099+0000", 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063462999999999996 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348122+0000", 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2866000000000001e-05 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356742+0000", 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086204220000000008 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356825+0000", 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2687000000000003e-05 2024-09-16T02:14:46.906 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.9s6 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.500357791999999, 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0076692430000000001, 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.907 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349127+0000", 2024-09-16T02:14:46.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1560999999999997e-05 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349203+0000", 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6447999999999997e-05 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349306+0000", 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102413 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349329+0000", 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349580+0000", 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000250874 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349604+0000", 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4253e-05 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356660+0000", 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070559610000000004 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356754+0000", 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4114e-05 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 45/32 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=45'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.498170503000001, 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010034855000000001, 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:14:46.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351322+0000", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351430+0000", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108572 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351458+0000", 2024-09-16T02:14:46.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8229e-05 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351962+0000", 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050377899999999999 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351989+0000", 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6322e-05 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361119+0000", 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091308310000000007 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361307+0000", 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188054 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 45/32 ECSubWrite(tid=22, reqid=osd.6.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.352169+0000", 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.497274096, 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047614170000000004, 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352169+0000", 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352169+0000", 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352171+0000", 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6270000000000001e-06 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352194+0000", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3388999999999999e-05 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352355+0000", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016054400000000001 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352379+0000", 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4613e-05 2024-09-16T02:14:46.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352562+0000", 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018259999999999999 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.352577+0000", 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4895999999999999e-05 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356860+0000", 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042835970000000001 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356930+0000", 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0151e-05 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.9s6 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.360173+0000", 2024-09-16T02:14:46.921 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.489270028, 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042969769999999996, 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360173+0000", 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360173+0000", 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360175+0000", 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1229999999999998e-06 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360204+0000", 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8476e-05 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360295+0000", 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1187000000000006e-05 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360318+0000", 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2829000000000001e-05 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360474+0000", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000155888 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360498+0000", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4256e-05 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364358+0000", 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003860455 2024-09-16T02:14:46.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364470+0000", 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011176299999999999 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.360354+0000", 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.489089454, 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0042497949999999998, 2024-09-16T02:14:46.926 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360354+0000", 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360354+0000", 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360355+0000", 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8959999999999999e-06 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360409+0000", 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3489000000000002e-05 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360479+0000", 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9640000000000004e-05 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360521+0000", 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2222000000000003e-05 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360708+0000", 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000186763 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360731+0000", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3062999999999999e-05 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364524+0000", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037930009999999998 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364603+0000", 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9721000000000005e-05 2024-09-16T02:14:46.930 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.6s5 51/32 ECSubWrite(tid=23, reqid=osd.15.0:23, at_version=51'14, trim_to=0'0, roll_forward_to=49'12))", 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.404661+0000", 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.4447819849999997, 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041513189999999997, 2024-09-16T02:14:46.931 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404661+0000", 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404661+0000", 2024-09-16T02:14:46.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404662+0000", 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9999999999999998e-07 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404744+0000", 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2155000000000002e-05 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404867+0000", 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012332900000000001 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.404895+0000", 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7922000000000001e-05 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.405073+0000", 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000178421 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.405101+0000", 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7813e-05 2024-09-16T02:14:46.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.408711+0000", 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0036102920000000002 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.408812+0000", 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000100887 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.936 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 51/32 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=51'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.4358396180000002, 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075541130000000003, 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413604+0000", 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:14:46.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413653+0000", 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8726000000000001e-05 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414322+0000", 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066915500000000005 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414345+0000", 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2863000000000001e-05 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414490+0000", 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014519199999999999 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414515+0000", 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5312000000000001e-05 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421073+0000", 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065573719999999997 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421158+0000", 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4690000000000004e-05 2024-09-16T02:14:46.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:394 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.942122543, 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094620629999999997, 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 394 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907323+0000", 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5399999999999998e-06 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907336+0000", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2605999999999999e-05 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907364+0000", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8538999999999999e-05 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907387+0000", 2024-09-16T02:14:46.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979000000000001e-05 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907499+0000", 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011208 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907666+0000", 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016685800000000001 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.909169+0000", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015026659999999999 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914888+0000", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057194580000000002 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916546+0000", 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001657911 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916783+0000", 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236426 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:395 3.fs0 3:f97ec54c:::smithi19530025-26:head [rollback 8] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:14:46.948 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.9420182329999998, 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011190179, 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 395 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907426+0000", 2024-09-16T02:14:46.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907433+0000", 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907453+0000", 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907485+0000", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2066000000000001e-05 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910164+0000", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026789510000000002 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910300+0000", 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013539300000000001 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.911564+0000", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001264577 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.917774+0000", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062099729999999997 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918506+0000", 2024-09-16T02:14:46.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073229000000000002 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918615+0000", 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108527 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:396 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.9419321790000001, 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011749021, 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 396 2024-09-16T02:14:46.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:46.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907512+0000", 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1060000000000001e-06 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907519+0000", 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.917e-06 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907537+0000", 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907552+0000", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912502+0000", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049500600000000001 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912741+0000", 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023893499999999999 2024-09-16T02:14:46.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914057+0000", 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013153119999999999 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918884+0000", 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048277240000000003 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919157+0000", 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306100000000001 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919260+0000", 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102594 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:46.960 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:46.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:46.962 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:46.958+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:46.964 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:14:46.964 INFO:tasks.rados.rados.0.smithi195.stdout:186: done (2 left) 2024-09-16T02:14:46.964 INFO:tasks.rados.rados.0.smithi195.stdout:187: done (1 left) 2024-09-16T02:14:46.964 INFO:tasks.rados.rados.0.smithi195.stdout:188: done (0 left) 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:189: setattr oid 33 current snap is 13 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:190: delete oid 6 current snap is 13 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:190: done (1 left) 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:191: rmattr oid 7 current snap is 13 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:191: done (1 left) 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:192: rollback oid 46 current snap is 13 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 46 to 11 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:193: read oid 7 snap -1 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:193: expect (ObjNum 89 snap 12 seq_num 89) 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:194: rmattr oid 21 current snap is 13 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:195: read oid 11 snap -1 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:195: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:14:46.965 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 45 current snap is 13 2024-09-16T02:14:46.966 INFO:tasks.rados.rados.0.smithi195.stdout:196: seq_num 92 ranges {1261568=786432,2048000=491520,2539520=278528} 2024-09-16T02:14:46.994 INFO:tasks.rados.rados.0.smithi195.stdout:196: writing smithi19530025-45 from 1261568 to 2048000 tid 1 2024-09-16T02:14:47.002 INFO:tasks.rados.rados.0.smithi195.stdout:196: writing smithi19530025-45 from 2048000 to 2539520 tid 2 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:196: writing smithi19530025-45 from 2539520 to 2818048 tid 3 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:197: snap_create 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 33 v 29 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:192: finishing rollback tid 0 to smithi19530025-46 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:192: finishing rollback tid 1 to smithi19530025-46 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:192: finishing rollback tid 2 to smithi19530025-46 2024-09-16T02:14:47.006 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 44 (ObjNum 177 snap 0 seq_num 1) dirty exists 2024-09-16T02:14:47.007 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 21 v 20 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:14:47.007 INFO:tasks.rados.rados.0.smithi195.stdout:193: expect (ObjNum 89 snap 12 seq_num 89) 2024-09-16T02:14:47.007 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:47.030 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:47.049 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:47.063 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:47.062+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:47.065 INFO:tasks.rados.rados.0.smithi195.stdout:196: finishing write tid 1 to smithi19530025-45 2024-09-16T02:14:47.065 INFO:tasks.rados.rados.0.smithi195.stdout:196: finishing write tid 2 to smithi19530025-45 2024-09-16T02:14:47.065 INFO:tasks.rados.rados.0.smithi195.stdout:196: finishing write tid 3 to smithi19530025-45 2024-09-16T02:14:47.065 INFO:tasks.rados.rados.0.smithi195.stdout:196: finishing write tid 4 to smithi19530025-45 2024-09-16T02:14:47.065 INFO:tasks.rados.rados.0.smithi195.stdout:196: finishing write tid 5 to smithi19530025-45 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:196: oid 45 updating version 0 to 37 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:196: oid 45 version 37 is already newer than 36 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:196: oid 45 version 37 is already newer than 35 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:196: oid 45 updating version 37 to 38 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 38 (ObjNum 92 snap 13 seq_num 92) dirty exists 2024-09-16T02:14:47.066 INFO:tasks.rados.rados.0.smithi195.stdout:196: left oid 45 (ObjNum 92 snap 13 seq_num 92) 2024-09-16T02:14:47.097 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:47.097 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:47.097 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:14:47.097 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:47.112 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:14:47.136 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:47.136 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:47.161 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:47.162 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:47.163 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:47.158+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:47.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:47.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:47.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:47.233 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:47.233 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:47.234 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:47.234 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:14:47.234 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:14:47.235 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:14:47.235 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:47.249 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:14:47.262 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:47.262 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:47.278 INFO:tasks.rados.rados.0.smithi195.stdout:195: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:14:47.283 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 284996 2024-09-16T02:14:47.290 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:47.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:47.293 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:47.291+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:47.325 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:47.325 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:47.350 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1658552 2024-09-16T02:14:47.352 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:47.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:47.365 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1451430 2024-09-16T02:14:47.365 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:47.358+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:47.373 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:47.373 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:47.385 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:47.385 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:47.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:14:47.410 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:47.410 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:47.435 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:47.435 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:47.459 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:47.460 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:47.466 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:47.466+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:47.485 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:47.486 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:47.514 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:47.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:47.545 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:47.545 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:47.554 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:47.554 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:47.567 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:47.563+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:47.568 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:14:47.669 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:47.663+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:47.691 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:47.691 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:47.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:14:47.769 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:47.767+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:47.822 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:47.822 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:47.836 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:14:47.870 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:47.862+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:47.941 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:14:47.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:14:47.941 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-16T02:14:47.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 11 2024-09-16T02:14:47.950 INFO:tasks.rados.rados.0.smithi195.stdout:189: done (6 left) 2024-09-16T02:14:47.950 INFO:tasks.rados.rados.0.smithi195.stdout:192: done (5 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:193: done (4 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:194: done (3 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:195: done (2 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:196: done (1 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:197: done (0 left) 2024-09-16T02:14:47.951 INFO:tasks.rados.rados.0.smithi195.stdout:198: setattr oid 14 current snap is 14 2024-09-16T02:14:47.952 INFO:tasks.rados.rados.0.smithi195.stdout:199: snap_remove snap 8 2024-09-16T02:14:47.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:14:47.971 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:47.963+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:48.005 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.006 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.021 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:14:48.071 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:48.067+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:48.144 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.144 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.158 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:14:48.172 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:48.170+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:48.273 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:48.267+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:48.274 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.275 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.283 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-09-16T02:14:48.278+0000 7f5f54a8b640 -1 mon.a@0(leader).osd e59 definitely_dead 0 2024-09-16T02:14:48.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:14:48.374 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:48.370+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:48.421 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.421 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:14:48.475 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:48.470+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:48.558 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.558 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:14:48.615 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:48.611+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:48.676 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:48.670+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:48.692 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.692 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.708 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:14:48.799 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:48.795+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:48.877 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:48.877 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:48.877 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:48.870+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:48.892 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:14:48.959 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.11. 2024-09-16T02:14:48.960 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:14:48.971 INFO:tasks.rados.rados.0.smithi195.stdout:199: done (1 left) 2024-09-16T02:14:48.971 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 48 current snap is 14 2024-09-16T02:14:48.971 INFO:tasks.rados.rados.0.smithi195.stdout:200: seq_num 93 ranges {0=802816,802816=638976,1441792=131072} 2024-09-16T02:14:48.973 INFO:tasks.rados.rados.0.smithi195.stdout:200: writing smithi19530025-48 from 0 to 802816 tid 1 2024-09-16T02:14:48.978 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:48.970+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:48.981 INFO:tasks.rados.rados.0.smithi195.stdout:200: writing smithi19530025-48 from 802816 to 1441792 tid 2 2024-09-16T02:14:48.982 INFO:tasks.rados.rados.0.smithi195.stdout:200: writing smithi19530025-48 from 1441792 to 1572864 tid 3 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 14 v 108 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:198: done (1 left) 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:201: rollback oid 4 current snap is 14 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 4 to 13 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:202: read oid 30 snap 12 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:202: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T02:14:48.983 INFO:tasks.rados.rados.0.smithi195.stdout:203: delete oid 14 current snap is 14 2024-09-16T02:14:48.997 INFO:tasks.rados.rados.0.smithi195.stdout:201: finishing rollback tid 1 to smithi19530025-4 2024-09-16T02:14:48.997 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 4 v 0 (ObjNum 289 snap 0 seq_num 3651708705) dirty dne 2024-09-16T02:14:49.003 INFO:tasks.rados.rados.0.smithi195.stdout:202: expect (ObjNum 51 snap 1 seq_num 51) 2024-09-16T02:14:49.004 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:49.024 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 472890 2024-09-16T02:14:49.042 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 793228 2024-09-16T02:14:49.075 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.075 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.078 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:49.071+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:49.090 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:14:49.179 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:49.175+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:49.253 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.253 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.267 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:14:49.280 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:49.278+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:49.381 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:49.374+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:49.433 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.434 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:14:49.481 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:49.479+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:49.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:49.571 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.572 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.581 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:49.579+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:49.586 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:14:49.682 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:49.678+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:49.707 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.707 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.721 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:14:49.741 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:49.741 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:49.767 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:49.767 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:49.784 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:49.779+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:49.793 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:49.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:49.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:49.839 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.839 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.850 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:49.850 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:49.854 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:14:49.881 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:49.881 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:49.885 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:49.883+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:49.914 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:49.914 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:49.944 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:49.977 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:49.977 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:49.981 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:49.981 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:49.986 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:49.978+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:49.992 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:14:50.016 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:50.017 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:50.056 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:50.057 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:50.085 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:50.085 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:50.087 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:50.083+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:50.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:50.109 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:50.110 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:50.110 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:50.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:50.142 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:50.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:50.167 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:50.167 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:50.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:50.162+0000 7f085ffff640 0 -- 172.21.15.3:0/1747684775 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f0860227bd0 msgr2=0x7f0860229fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:50.187 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:50.183+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:50.247 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:50.247 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:50.247 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:50.247 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:50.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:50.302 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:50.298+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:50.391 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:50.390+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:50.402 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:50.402 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:50.430 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:50.430 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:50.431 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:50.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:50.491 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:50.487+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:50.592 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:50.591+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:50.607 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:43 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'25, mlcod=24'20)", 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 44.025835358000002, 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055114170000000002, 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578971+0000", 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578972+0000", 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0299999999999998e-07 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.578977+0000", 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6269999999999997e-06 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579012+0000", 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4903000000000001e-05 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579023+0000", 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0949e-05 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579077+0000", 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3783000000000003e-05 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.579092+0000", 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5566999999999999e-05 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584412+0000", 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053199859999999996 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.584482+0000", 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9899000000000003e-05 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:74 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'52, mlcod=24'45)", 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.919616112, 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050466110000000003, 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685190+0000", 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685192+0000", 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5400000000000001e-06 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685202+0000", 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0287e-05 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685261+0000", 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8758999999999998e-05 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.685273+0000", 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.261e-05 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686230+0000", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095673999999999998 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686263+0000", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3319999999999999e-05 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.690175+0000", 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039115419999999996 2024-09-16T02:14:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.690237+0000", 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1812999999999999e-05 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:75 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'53, mlcod=24'46)", 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.918565692999998, 2024-09-16T02:14:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060039550000000001, 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686241+0000", 2024-09-16T02:14:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686242+0000", 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.564e-06 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686348+0000", 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105983 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686445+0000", 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6599000000000001e-05 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686462+0000", 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7557000000000001e-05 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.686971+0000", 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050895900000000002 2024-09-16T02:14:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687026+0000", 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4799999999999997e-05 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.692166+0000", 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051400029999999998 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.692245+0000", 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8490000000000002e-05 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:76 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'54, mlcod=24'46)", 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.917288067999998, 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007242999, 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687520+0000", 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.796e-06 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687531+0000", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1420000000000001e-05 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687607+0000", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5166999999999994e-05 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687620+0000", 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3873000000000001e-05 2024-09-16T02:14:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689759+0000", 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021384210000000002 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689788+0000", 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8782999999999999e-05 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694680+0000", 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004892123 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694761+0000", 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1415999999999994e-05 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:95 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'69, mlcod=24'65)", 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.799572695000002, 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057000499999999999, 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805234+0000", 2024-09-16T02:14:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805235+0000", 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9999999999999995e-07 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805244+0000", 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1260000000000004e-06 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805294+0000", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.02e-05 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805306+0000", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.217e-05 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805406+0000", 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9481999999999994e-05 2024-09-16T02:14:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.805441+0000", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5247000000000003e-05 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810846+0000", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054056449999999997 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810934+0000", 2024-09-16T02:14:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7180000000000002e-05 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:96 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'70, mlcod=24'65)", 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.798438746999999, 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063466429999999999, 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806369+0000", 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.297e-06 2024-09-16T02:14:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806477+0000", 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107983 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806532+0000", 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5075999999999999e-05 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806544+0000", 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2405999999999999e-05 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806623+0000", 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8689999999999994e-05 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806656+0000", 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3102999999999998e-05 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812648+0000", 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059919650000000001 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812714+0000", 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6123000000000004e-05 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:97 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'71, mlcod=24'66)", 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.796298618000002, 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056618140000000003, 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808508+0000", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808509+0000", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.823e-06 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808521+0000", 2024-09-16T02:14:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1759999999999999e-05 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808597+0000", 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5833000000000004e-05 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.808615+0000", 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8292999999999999e-05 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810373+0000", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001757315 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810436+0000", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2846000000000001e-05 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814082+0000", 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036462980000000001 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814169+0000", 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7645999999999994e-05 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:98 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'72, mlcod=24'67)", 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.794859958000004, 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005045051, 2024-09-16T02:14:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809946+0000", 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809948+0000", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.46e-06 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.809958+0000", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0596e-05 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810018+0000", 2024-09-16T02:14:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9833000000000001e-05 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.810029+0000", 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029999999999999e-05 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.811011+0000", 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098175699999999999 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.811066+0000", 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4765999999999997e-05 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814911+0000", 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003844737 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.814991+0000", 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0871999999999994e-05 2024-09-16T02:14:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:104 1.0 e24/23 1:30306672:devicehealth::main.db.0000000000000000:head v 24'78, mlcod=24'73)", 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.784531563999998, 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050841580000000001, 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820275+0000", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820276+0000", 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:14:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820326+0000", 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0062999999999998e-05 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820380+0000", 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4562999999999999e-05 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820390+0000", 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8970000000000003e-06 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820465+0000", 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4332999999999995e-05 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.820494+0000", 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9272999999999999e-05 2024-09-16T02:14:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.825267+0000", 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047728459999999999 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.825359+0000", 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2152999999999997e-05 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:116 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'86, mlcod=24'80)", 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.692939850999998, 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005826726, 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911866+0000", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911868+0000", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3400000000000001e-06 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.911878+0000", 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0699999999999999e-05 2024-09-16T02:14:50.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913929+0000", 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020504820000000002 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.913942+0000", 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.327e-05 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.914088+0000", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000145642 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.914122+0000", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4597000000000001e-05 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.917632+0000", 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035092779999999998 2024-09-16T02:14:50.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.917693+0000", 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1416999999999998e-05 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:122 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'92, mlcod=24'87)", 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.683684528999997, 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055942539999999999, 2024-09-16T02:14:50.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921122+0000", 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921125+0000", 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7999999999999999e-06 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.921139+0000", 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4912999999999999e-05 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922440+0000", 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013005670000000001 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922453+0000", 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2476999999999999e-05 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922551+0000", 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8579999999999995e-05 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922584+0000", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3172e-05 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.926588+0000", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040037329999999998 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.926716+0000", 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012801199999999999 2024-09-16T02:14:50.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'87)", 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.682491644999999, 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069325319999999999, 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922316+0000", 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4640000000000001e-06 2024-09-16T02:14:50.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922381+0000", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4969999999999996e-05 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924466+0000", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002085075 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924479+0000", 2024-09-16T02:14:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3033e-05 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924623+0000", 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924687+0000", 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3737000000000003e-05 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929168+0000", 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044810020000000004 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929247+0000", 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9472000000000001e-05 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'89)", 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.680385934999997, 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33740482100000002, 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924422+0000", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924431+0000", 2024-09-16T02:14:50.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8769999999999999e-06 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260013+0000", 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33558222100000001 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260031+0000", 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8063e-05 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260228+0000", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000196782 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260273+0000", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877000000000001e-05 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261745+0000", 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001471916 2024-09-16T02:14:50.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261825+0000", 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9791999999999998e-05 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.247801192000001, 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071696049999999999, 2024-09-16T02:14:50.668 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357006+0000", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9399999999999995e-07 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357033+0000", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6978000000000002e-05 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357063+0000", 2024-09-16T02:14:50.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0549999999999997e-05 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357074+0000", 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1077000000000001e-05 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357740+0000", 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066576700000000005 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357753+0000", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2706e-05 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364120+0000", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063668939999999997 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364175+0000", 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5139000000000002e-05 2024-09-16T02:14:50.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.245064274000001, 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068742350000000002, 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:14:50.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0699999999999998e-07 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359769+0000", 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6655000000000001e-05 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359836+0000", 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7043000000000002e-05 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359848+0000", 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.199e-05 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359928+0000", 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9927999999999998e-05 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359940+0000", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.202e-05 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366543+0000", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066029269999999998 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366616+0000", 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3264999999999997e-05 2024-09-16T02:14:50.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.244897420000001, 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067033939999999997, 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.678 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359910+0000", 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.22e-06 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359939+0000", 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9213999999999998e-05 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359987+0000", 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7602999999999998e-05 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359999+0000", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1666e-05 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360077+0000", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8336000000000004e-05 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360089+0000", 2024-09-16T02:14:50.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2432999999999999e-05 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366574+0000", 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064847710000000003 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366612+0000", 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8151000000000001e-05 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.es5 45/32 ECSubWrite(tid=17, reqid=osd.8.0:17, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.360297+0000", 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.244509718, 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063033339999999998, 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360297+0000", 2024-09-16T02:14:50.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360297+0000", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360297+0000", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4000000000000002e-07 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360322+0000", 2024-09-16T02:14:50.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4641e-05 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360377+0000", 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5736999999999998e-05 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360384+0000", 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9269999999999999e-06 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360461+0000", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6818999999999998e-05 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360470+0000", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8349999999999993e-06 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366548+0000", 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060779370000000003 2024-09-16T02:14:50.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366600+0000", 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1897999999999997e-05 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.242727946999999, 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080847929999999998, 2024-09-16T02:14:50.687 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:14:50.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362079+0000", 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.054e-06 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362106+0000", 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6322e-05 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362152+0000", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6276999999999998e-05 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362163+0000", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.064e-05 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362225+0000", 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2539999999999994e-05 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362236+0000", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1151999999999999e-05 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370090+0000", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00785368 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370163+0000", 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3127999999999995e-05 2024-09-16T02:14:50.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s4 52/32 ECSubWrite(tid=39, reqid=client.4598.0:402, at_version=52'21, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.956855+0000", 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 9.6479508880000004, 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048879250000000004, 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956855+0000", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956855+0000", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956857+0000", 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.576e-06 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956891+0000", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4412999999999997e-05 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956972+0000", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0909999999999996e-05 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.956987+0000", 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5036e-05 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957137+0000", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014976900000000001 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957166+0000", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8745999999999999e-05 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.961674+0000", 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045077980000000004 2024-09-16T02:14:50.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.961743+0000", 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9677000000000004e-05 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 52/32 ECSubWrite(tid=226, reqid=client.4598.0:412, at_version=52'47, trim_to=0'0, roll_forward_to=51'45))", 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.993094+0000", 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 9.6117118389999998, 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055210809999999997, 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:50.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993094+0000", 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993094+0000", 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993095+0000", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9000000000000002e-07 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993595+0000", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049992999999999999 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993661+0000", 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6519999999999993e-05 2024-09-16T02:14:50.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993678+0000", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.664e-05 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993836+0000", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015809900000000001 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.993856+0000", 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769e-05 2024-09-16T02:14:50.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.998518+0000", 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004662233 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.998616+0000", 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7499999999999998e-05 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:50.700 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:50.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:50.702 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:50.702+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:50.803 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:50.799+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:50.832 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:50.832 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:50.832 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:50.833 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:50.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:50.904 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:50.903+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:50.979 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:50.979 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:50.979 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:50.980 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:50.980 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:50.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:50.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:14:51.005 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:51.003+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:51.022 INFO:tasks.rados.rados.0.smithi195.stdout:200: finishing write tid 1 to smithi19530025-48 2024-09-16T02:14:51.023 INFO:tasks.rados.rados.0.smithi195.stdout:200: finishing write tid 2 to smithi19530025-48 2024-09-16T02:14:51.025 INFO:tasks.rados.rados.0.smithi195.stdout:200: finishing write tid 3 to smithi19530025-48 2024-09-16T02:14:51.029 INFO:tasks.rados.rados.0.smithi195.stdout:200: finishing write tid 4 to smithi19530025-48 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: finishing write tid 5 to smithi19530025-48 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: oid 48 updating version 0 to 109 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: oid 48 updating version 109 to 111 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: oid 48 updating version 111 to 112 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: oid 48 version 112 is already newer than 110 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 112 (ObjNum 93 snap 14 seq_num 93) dirty exists 2024-09-16T02:14:51.036 INFO:tasks.rados.rados.0.smithi195.stdout:200: left oid 48 (ObjNum 93 snap 14 seq_num 93) 2024-09-16T02:14:51.106 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:51.103+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:51.126 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.126 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:51.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:51.126 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:51.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 23.804761325000001, 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001906184, 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318698+0000", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318699+0000", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3939999999999999e-06 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318830+0000", 2024-09-16T02:14:51.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013092799999999999 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318876+0000", 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6335999999999998e-05 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318883+0000", 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1550000000000001e-06 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.129 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318984+0000", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100424 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.318995+0000", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0835e-05 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.320576+0000", 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001581432 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.130 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:27.320604+0000", 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7679999999999999e-05 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.777124583999999, 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003304399, 2024-09-16T02:14:51.131 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.132 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346336+0000", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8929999999999999e-06 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346359+0000", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2534000000000001e-05 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346435+0000", 2024-09-16T02:14:51.133 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6236000000000007e-05 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346451+0000", 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5707000000000001e-05 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346735+0000", 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028424199999999998 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.134 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346747+0000", 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1725999999999999e-05 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349591+0000", 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002844329 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349639+0000", 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7732000000000003e-05 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.775615415000001, 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039104070000000003, 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.136 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347844+0000", 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9999999999999999e-07 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.137 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347874+0000", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0167000000000001e-05 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347952+0000", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8427e-05 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347965+0000", 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2937e-05 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.138 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348056+0000", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0390999999999996e-05 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348066+0000", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0584000000000001e-05 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351696+0000", 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003630084 2024-09-16T02:14:51.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351754+0000", 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7516999999999998e-05 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.773837105999998, 2024-09-16T02:14:51.140 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046999930000000004, 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:51.141 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8400000000000001e-07 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349656+0000", 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4477999999999998e-05 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.142 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349726+0000", 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9680999999999999e-05 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349739+0000", 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.323e-05 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349814+0000", 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.143 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349826+0000", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1955e-05 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354271+0000", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044454020000000002 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354322+0000", 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0506000000000001e-05 2024-09-16T02:14:51.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.766227502, 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052360729999999999, 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.145 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357232+0000", 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.146 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357254+0000", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2855999999999999e-05 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5995000000000002e-05 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357322+0000", 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1798e-05 2024-09-16T02:14:51.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357409+0000", 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7282999999999998e-05 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357425+0000", 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.52e-05 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362424+0000", 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049990690000000001 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362467+0000", 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3674000000000002e-05 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:51.149 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.766095359000001, 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065655959999999999, 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.150 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357364+0000", 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8199999999999999e-07 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357381+0000", 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7022999999999999e-05 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.151 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357419+0000", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8779999999999998e-05 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357427+0000", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357549+0000", 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121402 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.152 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357561+0000", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2807e-05 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363880+0000", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063186550000000003 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363929+0000", 2024-09-16T02:14:51.153 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9030999999999999e-05 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.763848208999999, 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045504229999999996, 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.154 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359612+0000", 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.122e-06 2024-09-16T02:14:51.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359626+0000", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4422e-05 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359678+0000", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1765999999999999e-05 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359690+0000", 2024-09-16T02:14:51.156 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2143e-05 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359766+0000", 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5883000000000005e-05 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359776+0000", 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0448999999999999e-05 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.157 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364111+0000", 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043348809999999996 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364161+0000", 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9756999999999998e-05 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:14:51.158 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.763553310999999, 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065736739999999998, 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.159 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359931+0000", 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4975999999999999e-05 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359972+0000", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0244000000000003e-05 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359982+0000", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0710000000000001e-05 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360066+0000", 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3308999999999998e-05 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360080+0000", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4363e-05 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366432+0000", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00635214 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366479+0000", 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6959999999999998e-05 2024-09-16T02:14:51.162 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=25, reqid=osd.2.0:24, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.749344808, 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026568659999999999, 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374114+0000", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374115+0000", 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3850000000000001e-06 2024-09-16T02:14:51.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374134+0000", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8363999999999999e-05 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374183+0000", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9078e-05 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374195+0000", 2024-09-16T02:14:51.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1704e-05 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374270+0000", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999997e-05 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.374283+0000", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2044e-05 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.376722+0000", 2024-09-16T02:14:51.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024396600000000002 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.376771+0000", 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8721000000000003e-05 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=38, reqid=client.4598.0:394, at_version=52'23, trim_to=0'0, roll_forward_to=48'21))", 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:14:51.167 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.215607111000001, 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043643559999999998, 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:14:51.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907890+0000", 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7839999999999997e-05 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907996+0000", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106345 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908022+0000", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5293999999999999e-05 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908228+0000", 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206372 2024-09-16T02:14:51.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908273+0000", 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5225999999999997e-05 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912158+0000", 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038845490000000002 2024-09-16T02:14:51.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912216+0000", 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8359999999999997e-05 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=39, reqid=client.4598.0:395, at_version=52'24, trim_to=0'0, roll_forward_to=48'21))", 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.910467+0000", 2024-09-16T02:14:51.172 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.212991343000001, 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026172779999999998, 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910467+0000", 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910467+0000", 2024-09-16T02:14:51.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910469+0000", 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8640000000000001e-06 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910518+0000", 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8466e-05 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910619+0000", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100709 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.910642+0000", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.387e-05 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.911528+0000", 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000885507 2024-09-16T02:14:51.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.911554+0000", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5829999999999998e-05 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913004+0000", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014504400000000001 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913085+0000", 2024-09-16T02:14:51.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0592000000000004e-05 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=40, reqid=client.4598.0:396, at_version=52'25, trim_to=0'0, roll_forward_to=48'21))", 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.913128+0000", 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.210330932, 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019975560000000002, 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.177 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913128+0000", 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913128+0000", 2024-09-16T02:14:51.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913130+0000", 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7e-06 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913166+0000", 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6040000000000001e-05 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913256+0000", 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0478999999999999e-05 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913290+0000", 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3389999999999997e-05 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913429+0000", 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013981599999999999 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.913456+0000", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6270000000000001e-05 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.915011+0000", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015552560000000001 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.915125+0000", 2024-09-16T02:14:51.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114605 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=41, reqid=client.4598.0:396, at_version=0'0, trim_to=0'0, roll_forward_to=52'25))", 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.918456+0000", 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.205002982, 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002209964, 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918456+0000", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918456+0000", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918458+0000", 2024-09-16T02:14:51.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.813e-06 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918479+0000", 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1169999999999999e-05 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918589+0000", 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109726 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918604+0000", 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5886999999999999e-05 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918755+0000", 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015006199999999999 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.918774+0000", 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9962999999999998e-05 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.920573+0000", 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017989340000000001 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.920666+0000", 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2409000000000002e-05 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/32 ECSubWrite(tid=22, reqid=client.4598.0:404, at_version=52'7, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.966943+0000", 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.156515743, 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029999229999999998, 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966943+0000", 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966943+0000", 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966944+0000", 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6000000000000002e-07 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.967274+0000", 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033036799999999999 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.967406+0000", 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013127899999999999 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.967432+0000", 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6737000000000002e-05 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.967601+0000", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016844599999999999 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.967624+0000", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3676000000000002e-05 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.969846+0000", 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002221411 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.969943+0000", 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7145999999999995e-05 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/32 ECSubWrite(tid=23, reqid=client.4598.0:405, at_version=52'8, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.977878+0000", 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.145581213, 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024732819999999998, 2024-09-16T02:14:51.191 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977878+0000", 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977878+0000", 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977880+0000", 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.277e-06 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.978258+0000", 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037844000000000002 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.978363+0000", 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104653 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.978387+0000", 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4088999999999999e-05 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.978542+0000", 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015451899999999999 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.978558+0000", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6147e-05 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.980247+0000", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001689601 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.980351+0000", 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103556 2024-09-16T02:14:51.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 52/32 ECSubWrite(tid=24, reqid=client.4598.0:406, at_version=52'9, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.981227+0000", 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.142231877, 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025237319999999999, 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981227+0000", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981227+0000", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981229+0000", 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.767e-06 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981341+0000", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111952 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981422+0000", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1370000000000002e-05 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981438+0000", 2024-09-16T02:14:51.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5835999999999999e-05 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981629+0000", 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019115199999999999 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981655+0000", 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6027e-05 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.983617+0000", 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001962096 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.983751+0000", 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013353200000000001 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=54, reqid=client.4598.0:504, at_version=58'35, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:14:51.200 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.11127237, 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048877479999999999, 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012189+0000", 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0559999999999999e-06 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013086+0000", 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089770099999999999 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013357+0000", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027122399999999998 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013374+0000", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6160000000000001e-05 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013559+0000", 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185515 2024-09-16T02:14:51.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013585+0000", 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6037000000000002e-05 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016983+0000", 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033975289999999998 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017074+0000", 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1526000000000004e-05 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=55, reqid=client.4598.0:505, at_version=58'36, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.1103640800000001, 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042762490000000002, 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.205 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:14:51.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1399999999999997e-07 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013218+0000", 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012275200000000001 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013417+0000", 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019934400000000001 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013427+0000", 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3370000000000006e-06 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015391+0000", 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019645779999999998 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015399+0000", 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017304+0000", 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001904953 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017371+0000", 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6846000000000003e-05 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=56, reqid=client.4598.0:506, at_version=58'37, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:14:51.209 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.1102080509999999, 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056867990000000002, 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8999999999999997e-07 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013335+0000", 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4120000000000001e-05 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013545+0000", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020994500000000001 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013558+0000", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3044000000000001e-05 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015997+0000", 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024385710000000001 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016025+0000", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7932999999999998e-05 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018839+0000", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028140270000000002 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018938+0000", 2024-09-16T02:14:51.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8669e-05 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=57, reqid=client.4598.0:507, at_version=58'38, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.1098507509999997, 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059371409999999999, 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013609+0000", 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7999999999999995e-07 2024-09-16T02:14:51.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013650+0000", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1192000000000001e-05 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013688+0000", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7902999999999999e-05 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013698+0000", 2024-09-16T02:14:51.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9690000000000006e-06 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017129+0000", 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034309319999999998 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017136+0000", 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.396e-06 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019482+0000", 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002345813 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019545+0000", 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3355999999999994e-05 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.218 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:51.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:51.220 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:51.218+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:51.321 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:51.319+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:51.348 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.348 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:51.348 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:51.348 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:51.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:51.422 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:51.419+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:51.483 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.484 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:51.484 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:51.484 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:51.484 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:51.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:51.524 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:51.519+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:51.621 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.621 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.340047259999999, 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002207437, 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278077+0000", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278078+0000", 2024-09-16T02:14:51.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.15e-07 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278223+0000", 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014522 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278313+0000", 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9775e-05 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278329+0000", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6232999999999999e-05 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278450+0000", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120557 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.278461+0000", 2024-09-16T02:14:51.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1697e-05 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.280237+0000", 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001775151 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.280285+0000", 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8389e-05 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/32 ECSubWrite(tid=31, reqid=osd.3.0:31, at_version=44'28, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.270725829, 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049741270000000001, 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347399+0000", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347400+0000", 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3600000000000002e-07 2024-09-16T02:14:51.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347420+0000", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0194e-05 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347461+0000", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0744999999999997e-05 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347469+0000", 2024-09-16T02:14:51.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.96e-06 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347553+0000", 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4555000000000006e-05 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.347565+0000", 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1562999999999999e-05 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352277+0000", 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047123479999999999 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.352373+0000", 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5926000000000003e-05 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 44/32 ECSubWrite(tid=32, reqid=osd.3.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.2532733, 2024-09-16T02:14:51.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032542029999999998, 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364851+0000", 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364852+0000", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3799999999999999e-07 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364857+0000", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7769999999999999e-06 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364922+0000", 2024-09-16T02:14:51.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5023000000000004e-05 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.364932+0000", 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0078000000000001e-05 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.365047+0000", 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115016 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.365065+0000", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7011e-05 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.368063+0000", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029989909999999999 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:33.368106+0000", 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2169000000000002e-05 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.261739567999999, 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049611000000000004, 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356385+0000", 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9999999999999999e-07 2024-09-16T02:14:51.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356486+0000", 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100728 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356545+0000", 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8567999999999999e-05 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356559+0000", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3883000000000001e-05 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356630+0000", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1431000000000001e-05 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356642+0000", 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2418999999999999e-05 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.361297+0000", 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046547230000000004 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.361346+0000", 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9147999999999998e-05 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=34, reqid=osd.3.0:34, at_version=45'32, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.255544299, 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070140769999999996, 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362582+0000", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-06 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362610+0000", 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7945000000000001e-05 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362658+0000", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8439000000000002e-05 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362670+0000", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1960000000000001e-05 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362748+0000", 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8162000000000001e-05 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362759+0000", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0978999999999999e-05 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369332+0000", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065723960000000003 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369595+0000", 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026293399999999999 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.644 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(osd.11.22:2 3.es0 3:701f1021:::smithi19530025-33:head [assert-version v2,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.253143427000001, 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042977079999999999, 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364981+0000", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364984+0000", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1950000000000002e-06 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364993+0000", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2110000000000004e-06 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365019+0000", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6511000000000001e-05 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365031+0000", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1333e-05 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365119+0000", 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8095000000000003e-05 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.365234+0000", 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115127 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369279+0000", 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040452359999999998 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.250629527000001, 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050183099999999998, 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:51.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367495+0000", 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1799999999999999e-07 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.367880+0000", 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038417900000000001 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368076+0000", 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019593799999999999 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368085+0000", 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4749999999999999e-06 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368154+0000", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9239000000000006e-05 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.368166+0000", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2055999999999999e-05 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.372477+0000", 2024-09-16T02:14:51.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043104889999999998 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.372514+0000", 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6715999999999999e-05 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=35, reqid=osd.3.0:34, at_version=0'0, trim_to=0'0, roll_forward_to=45'32))", 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.240772372999999, 2024-09-16T02:14:51.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027598050000000002, 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377352+0000", 2024-09-16T02:14:51.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377353+0000", 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8200000000000001e-07 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377365+0000", 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2094e-05 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377467+0000", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102618 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377478+0000", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0879999999999999e-05 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377694+0000", 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021557799999999999 2024-09-16T02:14:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.377707+0000", 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3237999999999999e-05 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380056+0000", 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002349425 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380112+0000", 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5689999999999997e-05 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 45/32 ECSubWrite(tid=22, reqid=osd.10.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.237573379000001, 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022536359999999998, 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380551+0000", 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380552+0000", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.47e-07 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380562+0000", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.448e-06 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380602+0000", 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9894999999999997e-05 2024-09-16T02:14:51.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380612+0000", 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8649999999999996e-06 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380666+0000", 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4789e-05 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.380674+0000", 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.382761+0000", 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020867749999999999 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.382805+0000", 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4218000000000003e-05 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.6s3 52/32 ECSubWrite(tid=25, reqid=client.4598.0:397, at_version=52'15, trim_to=0'0, roll_forward_to=51'14))", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.924932+0000", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.693193087000001, 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002348087, 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.924932+0000", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.924932+0000", 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.924932+0000", 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8400000000000004e-07 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.925290+0000", 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035777399999999999 2024-09-16T02:14:51.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.925369+0000", 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8886000000000003e-05 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.925394+0000", 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5352999999999999e-05 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.925534+0000", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013965200000000001 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.925553+0000", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9296999999999998e-05 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.927197+0000", 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016437610000000001 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.927280+0000", 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2680000000000001e-05 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 52/32 ECSubWrite(tid=35, reqid=client.4598.0:399, at_version=52'18, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.941821+0000", 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.676303666000001, 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003385416, 2024-09-16T02:14:51.667 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.941821+0000", 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.941821+0000", 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.941822+0000", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2e-07 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.942326+0000", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050428699999999996 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.942388+0000", 2024-09-16T02:14:51.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2534999999999997e-05 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.942404+0000", 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5894000000000001e-05 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.942544+0000", 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013957499999999999 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.942560+0000", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5826999999999999e-05 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.945115+0000", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025557790000000002 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.945207+0000", 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1099000000000003e-05 2024-09-16T02:14:51.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 52/32 ECSubWrite(tid=38, reqid=client.4598.0:401, at_version=52'20, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.954865+0000", 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.663259318, 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028855389999999999, 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.672 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.954865+0000", 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.954865+0000", 2024-09-16T02:14:51.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.954867+0000", 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.954949+0000", 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2596000000000007e-05 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.955033+0000", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3707000000000003e-05 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.955057+0000", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3896e-05 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.955220+0000", 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016304199999999999 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.955246+0000", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5639999999999998e-05 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957688+0000", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024428459999999998 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957751+0000", 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2485999999999998e-05 2024-09-16T02:14:51.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 52/32 ECSubWrite(tid=39, reqid=client.4598.0:402, at_version=52'21, trim_to=0'0, roll_forward_to=52'18))", 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.957030+0000", 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.661095157, 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049276110000000001, 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.677 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957030+0000", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957030+0000", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957032+0000", 2024-09-16T02:14:51.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9130000000000001e-06 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957076+0000", 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4536000000000002e-05 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957167+0000", 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0699999999999996e-05 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957194+0000", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7645999999999999e-05 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957320+0000", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012593599999999999 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.957335+0000", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-05 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.961817+0000", 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044823340000000001 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.961957+0000", 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139856 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 56/32 ECSubWrite(tid=51, reqid=client.4598.0:476, at_version=56'10, trim_to=0'0, roll_forward_to=54'8))", 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:44.992521+0000", 2024-09-16T02:14:51.681 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.625603474, 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026708999999999999, 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.992521+0000", 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.992521+0000", 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.992521+0000", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.55e-07 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.993034+0000", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000512256 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.993097+0000", 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3242000000000002e-05 2024-09-16T02:14:51.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.993123+0000", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6546999999999998e-05 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.993232+0000", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010836300000000001 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.993245+0000", 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.288e-05 2024-09-16T02:14:51.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.995144+0000", 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018996950000000001 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:44.995192+0000", 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7762000000000001e-05 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 56/32 ECSubWrite(tid=58, reqid=client.4598.0:483, at_version=56'16, trim_to=0'0, roll_forward_to=56'13))", 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.009734+0000", 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.6083911960000004, 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024416749999999999, 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009734+0000", 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009734+0000", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009734+0000", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7300000000000005e-07 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009767+0000", 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2271000000000003e-05 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009805+0000", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8449999999999999e-05 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009815+0000", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3929999999999997e-06 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009890+0000", 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5538999999999996e-05 2024-09-16T02:14:51.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.009900+0000", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0332e-05 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.012133+0000", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022329149999999998 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.012175+0000", 2024-09-16T02:14:51.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1902e-05 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:488 3.es0 3:701f1021:::smithi19530025-33:head [read 395661~145011,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.685282763, 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080293050000000005, 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:51.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 488 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932845+0000", 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1089999999999998e-06 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932860+0000", 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4576000000000001e-05 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932891+0000", 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1504000000000002e-05 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932910+0000", 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8334000000000001e-05 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933050+0000", 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014058899999999999 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933068+0000", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7396000000000001e-05 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938059+0000", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049910120000000004 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938174+0000", 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115674 2024-09-16T02:14:51.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.940871+0000", 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026971109999999999 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:489 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.6851797079999997, 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092923889999999999, 2024-09-16T02:14:51.695 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 489 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932946+0000", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932970+0000", 2024-09-16T02:14:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932983+0000", 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.27e-05 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932991+0000", 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933148+0000", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156828 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933157+0000", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8409999999999995e-06 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938656+0000", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005498541 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:51.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938725+0000", 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9061999999999997e-05 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.942237+0000", 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035124739999999998 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:490 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:14:51.700 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.6850957979999999, 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010928416999999999, 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 490 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933030+0000", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6700000000000003e-07 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933034+0000", 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7169999999999999e-06 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933046+0000", 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1584e-05 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933060+0000", 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3953e-05 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933191+0000", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131347 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933199+0000", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7030000000000006e-06 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939244+0000", 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060455639999999998 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939311+0000", 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6313e-05 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.943957+0000", 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046465689999999997 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:492 3.es0 3:701f1021:::smithi19530025-33:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.6683271470000003, 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017209518, 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 492 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.706 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949801+0000", 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2100000000000002e-06 2024-09-16T02:14:51.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949954+0000", 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015366300000000001 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949991+0000", 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6463e-05 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950016+0000", 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4593e-05 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950156+0000", 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000140882 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950172+0000", 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5590000000000002e-05 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954336+0000", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041644309999999997 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954560+0000", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022320200000000001 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.956344+0000", 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017841000000000001 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.958842+0000", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024979220000000001 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.966762+0000", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079201440000000005 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.967007+0000", 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024531800000000001 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.6s3 58/32 ECSubWrite(tid=33, reqid=client.4598.0:493, at_version=58'19, trim_to=0'0, roll_forward_to=54'17))", 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.951861+0000", 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4.6662634609999998, 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024955390000000002, 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.951861+0000", 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.951861+0000", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.951862+0000", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5699999999999998e-07 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.951909+0000", 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6872000000000001e-05 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.951994+0000", 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5414000000000006e-05 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.952017+0000", 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3465999999999999e-05 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:51.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.952176+0000", 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015897900000000001 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.952198+0000", 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1688999999999999e-05 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954273+0000", 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020744499999999998 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:51.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954357+0000", 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4312000000000002e-05 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:51.716 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:51.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:200: done (3 left) 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:201: done (2 left) 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:202: done (1 left) 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:203: done (0 left) 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:204: read oid 2 snap -1 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:204: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-16T02:14:51.722 INFO:tasks.rados.rados.0.smithi195.stdout:205: read oid 48 snap -1 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:205: expect (ObjNum 93 snap 14 seq_num 93) 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:206: read oid 16 snap -1 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:206: expect deleted 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:207: rmattr oid 13 current snap is 14 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:207: done (3 left) 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:208: read oid 6 snap 13 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:208: expect deleted 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:209: snap_create 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:204: expect (ObjNum 72 snap 7 seq_num 72) 2024-09-16T02:14:51.723 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 799178 2024-09-16T02:14:51.723 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:14:51.711+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:14:51.737 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 748503 2024-09-16T02:14:51.760 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 890434 2024-09-16T02:14:51.766 INFO:tasks.rados.rados.0.smithi195.stdout:205: expect (ObjNum 93 snap 14 seq_num 93) 2024-09-16T02:14:51.766 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:51.787 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:51.804 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:51.819 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:51.819+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:51.854 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:51.854 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:14:51.854 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:51.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:14:51.919 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:51.915+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:51.987 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:14:51.996 INFO:tasks.rados.rados.0.smithi195.stdout:204: done (4 left) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:205: done (3 left) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:206: done (2 left) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:208: done (1 left) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:209: done (0 left) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:210: read oid 18 snap -1 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:210: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T02:14:51.997 INFO:tasks.rados.rados.0.smithi195.stdout:211: snap_create 2024-09-16T02:14:51.998 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:14:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:14:51.998 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:14:51.998 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:52.010 INFO:tasks.rados.rados.0.smithi195.stdout:210: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T02:14:52.010 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:52.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:14:52.015 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 75212 2024-09-16T02:14:52.016 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 18069 2024-09-16T02:14:52.058 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:52.054+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:52.120 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:52.114+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:46 3.0s0 3:0952f51a:::smithi19530025-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:52.180 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.876133201000002, 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013865147, 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300541+0000", 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.193e-06 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301465+0000", 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092351300000000005 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301478+0000", 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.348e-05 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301487+0000", 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3869999999999995e-06 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301559+0000", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2444e-05 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301744+0000", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185019 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.302843+0000", 2024-09-16T02:14:52.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010987550000000001 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305418+0000", 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025746219999999999 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314319+0000", 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089012259999999999 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314404+0000", 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5507999999999995e-05 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:47 3.0s0 3:0952f51a:::smithi19530025-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.875164941000001, 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013156407, 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.186 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 47 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301508+0000", 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079e-06 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301513+0000", 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.194e-06 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301521+0000", 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2309999999999996e-06 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301527+0000", 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4710000000000004e-06 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304040+0000", 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025124589999999999 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304279+0000", 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023908499999999999 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304869+0000", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058981500000000004 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.307149+0000", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022805849999999999 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314589+0000", 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074399310000000003 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314664+0000", 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:48 3.0s0 3:0952f51a:::smithi19530025-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.874979157999999, 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021785207000000001, 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 48 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0359999999999999e-06 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301701+0000", 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9700000000000002e-06 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301718+0000", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7201999999999999e-05 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301729+0000", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0516999999999999e-05 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305321+0000", 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035922570000000002 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305405+0000", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3998000000000005e-05 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314939+0000", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095341579999999992 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.315042+0000", 2024-09-16T02:14:52.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102858 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323478+0000", 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084362110000000008 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:139 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:52.197 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.760030764, 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066880840000000004, 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416644+0000", 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9860000000000001e-06 2024-09-16T02:14:52.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417552+0000", 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090839999999999996 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417565+0000", 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3406e-05 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417572+0000", 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2779999999999999e-06 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417622+0000", 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0717999999999998e-05 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417725+0000", 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102357 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.418463+0000", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073846399999999998 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419987+0000", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001523735 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423245+0000", 2024-09-16T02:14:52.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032577930000000001 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423330+0000", 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4946999999999998e-05 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:140 3.9s0 3:9a381d46:::smithi19530025-47:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.203 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.759085037999998, 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060159289999999997, 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417587+0000", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417588+0000", 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3200000000000005e-07 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417591+0000", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6800000000000002e-06 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417596+0000", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8150000000000002e-06 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417600+0000", 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.337e-06 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419092+0000", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014920630000000001 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419203+0000", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110753 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419707+0000", 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050430500000000003 2024-09-16T02:14:52.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.420177+0000", 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047057700000000002 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423523+0000", 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003345878 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423603+0000", 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9888999999999994e-05 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:141 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.759067278, 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073674559999999997, 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417608+0000", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8899999999999999e-06 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417612+0000", 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0770000000000001e-06 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417616+0000", 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1389999999999999e-06 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419930+0000", 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023145040000000002 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419980+0000", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0439000000000003e-05 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423911+0000", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039302770000000002 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424016+0000", 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104734 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424973+0000", 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095702600000000004 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:252 3.0s0 3:0952f51a:::smithi19530025-16:head [stat,delete] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.831745303000002, 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011932368, 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.214 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 252 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344929+0000", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2709999999999999e-06 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344944+0000", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-05 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344959+0000", 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4990999999999999e-05 2024-09-16T02:14:52.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344967+0000", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9670000000000001e-06 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345036+0000", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9412999999999994e-05 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345046+0000", 2024-09-16T02:14:52.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6390000000000004e-06 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346669+0000", 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016225289999999999 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346810+0000", 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141457 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347909+0000", 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010991670000000001 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349212+0000", 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013032880000000001 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356764+0000", 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007551978 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4977999999999998e-05 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.825505385, 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095540220000000006, 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351168+0000", 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7899999999999997e-07 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351191+0000", 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3377e-05 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351238+0000", 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351247+0000", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2310000000000002e-06 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351304+0000", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7058000000000002e-05 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351312+0000", 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8790000000000003e-06 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360676+0000", 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093640279999999999 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360721+0000", 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4975999999999998e-05 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.819070448000002, 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066953009999999999, 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357623+0000", 2024-09-16T02:14:52.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0469999999999999e-05 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357662+0000", 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9601999999999998e-05 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357670+0000", 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8320000000000006e-06 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357735+0000", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4970999999999998e-05 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357744+0000", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3899999999999999e-06 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364244+0000", 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064994440000000001 2024-09-16T02:14:52.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364297+0000", 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3393999999999997e-05 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs3 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.818395808999998, 2024-09-16T02:14:52.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081662019999999991, 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:14:52.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5699999999999998e-07 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358303+0000", 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6217999999999999e-05 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358461+0000", 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015794100000000001 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358471+0000", 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5030000000000003e-06 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358566+0000", 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5378999999999996e-05 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358578+0000", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254e-05 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366371+0000", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077921409999999998 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366443+0000", 2024-09-16T02:14:52.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2123e-05 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 45/32 ECSubWriteReply(tid=12, last_complete=45'3, committed=1, applied=1))", 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.815720038999999, 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82653599300000002, 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.234 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360954+0000", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42e-06 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360960+0000", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6340000000000003e-06 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187311+0000", 2024-09-16T02:14:52.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82635024400000001 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187328+0000", 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7880000000000002e-05 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187449+0000", 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012031000000000001 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187488+0000", 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9505000000000002e-05 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.814351987999999, 2024-09-16T02:14:52.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074648689999999998, 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:14:52.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362321+0000", 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8199999999999999e-07 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362342+0000", 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0996999999999999e-05 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362436+0000", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3445999999999999e-05 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362445+0000", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3950000000000007e-06 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362539+0000", 2024-09-16T02:14:52.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3527000000000001e-05 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362548+0000", 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9429999999999995e-06 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369727+0000", 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071787120000000003 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369785+0000", 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8066999999999998e-05 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:404 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~606208] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.214487735000001, 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022657539000000001, 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.243 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 404 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:14:52.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962187+0000", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8729999999999999e-06 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963628+0000", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001440113 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963678+0000", 2024-09-16T02:14:52.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0349999999999997e-05 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963702+0000", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4533e-05 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963873+0000", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170669 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.964084+0000", 2024-09-16T02:14:52.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021121200000000001 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966544+0000", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024596430000000001 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.969112+0000", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025675950000000002 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984664+0000", 2024-09-16T02:14:52.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015552145999999999 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984842+0000", 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178405 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:405 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~720896] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.202173309000001, 2024-09-16T02:14:52.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011549382, 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 405 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974502+0000", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975490+0000", 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098815699999999992 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975523+0000", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3583e-05 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975539+0000", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5330000000000001e-05 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975692+0000", 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015385600000000001 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975935+0000", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024296100000000001 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977706+0000", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017700909999999999 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984097+0000", 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063910879999999996 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985929+0000", 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018325940000000001 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986048+0000", 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119226 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:406 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~49152] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.20109987, 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011213219999999999, 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 406 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975574+0000", 2024-09-16T02:14:52.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4300000000000001e-06 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975631+0000", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7519999999999998e-05 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975655+0000", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3465999999999999e-05 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975671+0000", 2024-09-16T02:14:52.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6186999999999999e-05 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.979950+0000", 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042792400000000001 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.980113+0000", 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016220199999999999 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981185+0000", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001072092 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984325+0000", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031403780000000001 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986680+0000", 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023551359999999999 2024-09-16T02:14:52.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986786+0000", 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010556900000000001 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:407 3.9s0 3:9a381d46:::smithi19530025-47:head [setxattr _header (57) in=7b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.200971571, 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.35746389899999997, 2024-09-16T02:14:52.259 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 407 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975702+0000", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.57e-06 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975711+0000", 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2900000000000002e-06 2024-09-16T02:14:52.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975730+0000", 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9017000000000001e-05 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975749+0000", 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8850000000000001e-05 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982236+0000", 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064873109999999999 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982511+0000", 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027516800000000001 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.983460+0000", 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094918300000000004 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985743+0000", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00228254 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.332957+0000", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.347213828 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333165+0000", 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000208142 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.264 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:408 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.200901654999999, 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.36004730099999999, 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:52.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 408 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975772+0000", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.544e-06 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975779+0000", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2829999999999998e-06 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975801+0000", 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2082999999999998e-05 2024-09-16T02:14:52.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975816+0000", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5662999999999999e-05 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986954+0000", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011137461 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.987093+0000", 2024-09-16T02:14:52.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139436 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333971+0000", 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34687802400000001 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.334106+0000", 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013453200000000001 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.335818+0000", 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017122750000000001 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 58/32 ECSubWrite(tid=50, reqid=client.4598.0:494, at_version=58'42, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.963022+0000", 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.2136501580000001, 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054846920000000002, 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963022+0000", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963022+0000", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963023+0000", 2024-09-16T02:14:52.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7599999999999998e-07 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963068+0000", 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5287000000000002e-05 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963182+0000", 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114406 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963204+0000", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1596000000000001e-05 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963392+0000", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018777299999999999 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963429+0000", 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7181999999999997e-05 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.968449+0000", 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050201509999999996 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.968507+0000", 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7921000000000003e-05 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.274 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.2136006149999998, 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069953569999999998, 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3000000000000001e-07 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963092+0000", 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9803e-05 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963251+0000", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015900899999999999 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963269+0000", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8363e-05 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.965522+0000", 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022524340000000002 2024-09-16T02:14:52.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.965546+0000", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4485999999999999e-05 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.969996+0000", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044494419999999996 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.970067+0000", 2024-09-16T02:14:52.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1390000000000006e-05 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 58/32 ECSubWrite(tid=52, reqid=client.4598.0:496, at_version=58'44, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.964606+0000", 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.2120665759999998, 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062856960000000003, 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964606+0000", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964606+0000", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964608+0000", 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9829999999999999e-06 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964648+0000", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0599999999999998e-05 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964743+0000", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5103000000000001e-05 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.964772+0000", 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8656000000000001e-05 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.967747+0000", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029744519999999998 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.967764+0000", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7686999999999999e-05 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.970836+0000", 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003071926 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:14:52.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.970891+0000", 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5288999999999999e-05 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:14:52.283 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:14:52.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:14:52.286 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:52.278+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:52.386 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:52.383+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:52.417 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:52.417 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:52.417 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:14:52.417 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:52.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:52.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:14:52.486 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:52.486+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:52.548 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:14:52.548 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:14:52.548 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:14:52.548 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:14:52.548 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:14:52.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:14:52.588 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:52.583+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:52.603 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:52.604 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:52.628 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:52.629 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:52.654 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:52.654 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:52.679 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:52.689 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:52.687+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:52.707 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:52.708 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:52.730 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:52.730 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:52.738 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:52.738 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:52.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:14:52.768 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:52.768 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:52.791 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:52.790+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:52.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:52.801 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:52.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:52.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:52.864 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:52.867 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:52.867 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:52.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:14:52.891 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:52.886+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:52.897 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:52.898 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:52.932 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:52.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:52.965 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:52.965 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:52.992 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:52.987+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:53.000 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:53.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:53.006 INFO:tasks.rados.rados.0.smithi195.stdout:210: done (1 left) 2024-09-16T02:14:53.006 INFO:tasks.rados.rados.0.smithi195.stdout:211: done (0 left) 2024-09-16T02:14:53.007 INFO:tasks.rados.rados.0.smithi195.stdout:212: delete oid 46 current snap is 16 2024-09-16T02:14:53.023 INFO:tasks.rados.rados.0.smithi195.stdout:212: done (0 left) 2024-09-16T02:14:53.024 INFO:tasks.rados.rados.0.smithi195.stdout:213: read oid 42 snap 14 2024-09-16T02:14:53.024 INFO:tasks.rados.rados.0.smithi195.stdout:213: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T02:14:53.024 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 45 current snap is 16 2024-09-16T02:14:53.024 INFO:tasks.rados.rados.0.smithi195.stdout:214: seq_num 96 ranges {2818048=458752,3276800=688128,3964928=475136} 2024-09-16T02:14:53.038 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:53.034+0000 7f2892d9d640 0 -- 172.21.15.3:0/247067810 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f2894225430 msgr2=0x7f2894227880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:53.074 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.074 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.076 INFO:tasks.rados.rados.0.smithi195.stdout:214: writing smithi19530025-45 from 2818048 to 3276800 tid 1 2024-09-16T02:14:53.085 INFO:tasks.rados.rados.0.smithi195.stdout:214: writing smithi19530025-45 from 3276800 to 3964928 tid 2 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:214: writing smithi19530025-45 from 3964928 to 4440064 tid 3 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:215: read oid 10 snap 14 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:215: expect (ObjNum 77 snap 8 seq_num 77) 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:216: rmattr oid 30 current snap is 16 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:216: done (3 left) 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 17 current snap is 16 2024-09-16T02:14:53.091 INFO:tasks.rados.rados.0.smithi195.stdout:217: seq_num 97 ranges {1523712=475136,1998848=442368,2441216=442368,2883584=278528} 2024-09-16T02:14:53.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:14:53.093 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:53.091+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:53.108 INFO:tasks.rados.rados.0.smithi195.stdout:217: writing smithi19530025-17 from 1523712 to 1998848 tid 1 2024-09-16T02:14:53.112 INFO:tasks.rados.rados.0.smithi195.stdout:217: writing smithi19530025-17 from 1998848 to 2441216 tid 2 2024-09-16T02:14:53.115 INFO:tasks.rados.rados.0.smithi195.stdout:217: writing smithi19530025-17 from 2441216 to 2883584 tid 3 2024-09-16T02:14:53.118 INFO:tasks.rados.rados.0.smithi195.stdout:217: writing smithi19530025-17 from 2883584 to 3162112 tid 4 2024-09-16T02:14:53.118 INFO:tasks.rados.rados.0.smithi195.stdout:218: copy_from oid 23 from oid 22 current snap is 16 2024-09-16T02:14:53.118 INFO:tasks.rados.rados.0.smithi195.stdout:213: expect (ObjNum 41 snap 0 seq_num 41) 2024-09-16T02:14:53.118 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.131 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.143 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.153 INFO:tasks.rados.rados.0.smithi195.stdout:214: finishing write tid 1 to smithi19530025-45 2024-09-16T02:14:53.153 INFO:tasks.rados.rados.0.smithi195.stdout:214: finishing write tid 2 to smithi19530025-45 2024-09-16T02:14:53.153 INFO:tasks.rados.rados.0.smithi195.stdout:214: finishing write tid 3 to smithi19530025-45 2024-09-16T02:14:53.153 INFO:tasks.rados.rados.0.smithi195.stdout:214: finishing write tid 4 to smithi19530025-45 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: finishing write tid 5 to smithi19530025-45 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: oid 45 updating version 0 to 44 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: oid 45 version 44 is already newer than 42 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: oid 45 updating version 44 to 45 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: oid 45 version 45 is already newer than 43 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 45 (ObjNum 96 snap 16 seq_num 96) dirty exists 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:214: left oid 45 (ObjNum 96 snap 16 seq_num 96) 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:215: expect (ObjNum 77 snap 8 seq_num 77) 2024-09-16T02:14:53.154 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.174 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.191 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:53.194 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:53.191+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:53.206 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 1 to smithi19530025-17 2024-09-16T02:14:53.206 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 2 to smithi19530025-17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 3 to smithi19530025-17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 4 to smithi19530025-17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 5 to smithi19530025-17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:218: finishing copy_from racing read to smithi19530025-23 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: finishing write tid 6 to smithi19530025-17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: oid 17 updating version 0 to 20 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: oid 17 version 20 is already newer than 19 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: oid 17 version 20 is already newer than 18 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: oid 17 version 20 is already newer than 17 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: oid 17 updating version 20 to 21 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 17 v 21 (ObjNum 97 snap 16 seq_num 97) dirty exists 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:217: left oid 17 (ObjNum 97 snap 16 seq_num 97) 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:218: finishing copy_from to smithi19530025-23 2024-09-16T02:14:53.207 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 23 v 8 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:213: done (4 left) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:214: done (3 left) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:215: done (2 left) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:217: done (1 left) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:218: done (0 left) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:219: read oid 5 snap -1 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:219: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 46 current snap is 16 2024-09-16T02:14:53.208 INFO:tasks.rados.rados.0.smithi195.stdout:220: seq_num 98 ranges {0=491520,491520=589824,1081344=557056,1638400=16384} 2024-09-16T02:14:53.211 INFO:tasks.rados.rados.0.smithi195.stdout:220: writing smithi19530025-46 from 0 to 491520 tid 1 2024-09-16T02:14:53.216 INFO:tasks.rados.rados.0.smithi195.stdout:220: writing smithi19530025-46 from 491520 to 1081344 tid 2 2024-09-16T02:14:53.218 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.218 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.222 INFO:tasks.rados.rados.0.smithi195.stdout:220: writing smithi19530025-46 from 1081344 to 1638400 tid 3 2024-09-16T02:14:53.222 INFO:tasks.rados.rados.0.smithi195.stdout:220: writing smithi19530025-46 from 1638400 to 1654784 tid 4 2024-09-16T02:14:53.223 INFO:tasks.rados.rados.0.smithi195.stdout:221: read oid 45 snap -1 2024-09-16T02:14:53.223 INFO:tasks.rados.rados.0.smithi195.stdout:221: expect (ObjNum 96 snap 16 seq_num 96) 2024-09-16T02:14:53.223 INFO:tasks.rados.rados.0.smithi195.stdout:222: snap_remove snap 15 2024-09-16T02:14:53.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:14:53.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:53.294 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:53.290+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:53.356 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.356 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:14:53.395 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:53.390+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:53.496 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:53.494+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:53.535 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.536 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.549 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:14:53.596 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:53.590+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:53.697 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:53.695+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:53.716 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.716 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:14:53.798 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:53.795+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:53.850 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.850 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:14:53.899 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:53.898+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:53.981 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:53.981 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:53.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:14:53.998 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15] out_osds: [11] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:14:53.998 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:14:53.999 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-09-16T02:14:53.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 11 2024-09-16T02:14:53.999 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:53.994+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:54.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:14:54.014 INFO:tasks.rados.rados.0.smithi195.stdout:219: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:14:54.026 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1245414 2024-09-16T02:14:54.031 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:54.054 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:54.075 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 1 to smithi19530025-46 2024-09-16T02:14:54.075 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 2 to smithi19530025-46 2024-09-16T02:14:54.076 INFO:tasks.rados.rados.0.smithi195.stdout:219: done (3 left) 2024-09-16T02:14:54.076 INFO:tasks.rados.rados.0.smithi195.stdout:222: done (2 left) 2024-09-16T02:14:54.076 INFO:tasks.rados.rados.0.smithi195.stdout:223: snap_remove snap 14 2024-09-16T02:14:54.101 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:54.099+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:54.116 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.116 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.130 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:14:54.201 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:54.198+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:54.263 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.263 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.278 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:14:54.302 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:54.302+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:54.358 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-09-16T02:14:54.354+0000 7f5f54a8b640 -1 mon.a@0(leader).osd e65 definitely_dead 0 2024-09-16T02:14:54.395 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.396 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.403 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:54.399+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:54.409 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:14:54.504 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:54.499+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:54.574 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.575 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.588 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:14:54.605 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:54.602+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:54.706 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:54.702+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:54.757 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.757 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.771 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:14:54.817 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:54.815+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:54.888 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:54.889 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:54.902 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:14:54.945 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:54.943+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:55.009 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.11. 2024-09-16T02:14:55.009 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:14:55.010 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:55.006+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:55.016 INFO:tasks.rados.rados.0.smithi195.stdout:223: done (2 left) 2024-09-16T02:14:55.016 INFO:tasks.rados.rados.0.smithi195.stdout:224: copy_from oid 8 from oid 42 current snap is 16 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 3 to smithi19530025-46 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 4 to smithi19530025-46 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 5 to smithi19530025-46 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:225: rmattr oid 28 current snap is 16 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:225: done (3 left) 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:226: copy_from oid 47 from oid 5 current snap is 16 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 37 current snap is 16 2024-09-16T02:14:55.017 INFO:tasks.rados.rados.0.smithi195.stdout:227: seq_num 99 ranges {1736704=507904,2244608=606208,2850816=557056} 2024-09-16T02:14:55.036 INFO:tasks.rados.rados.0.smithi195.stdout:227: writing smithi19530025-37 from 1736704 to 2244608 tid 1 2024-09-16T02:14:55.045 INFO:tasks.rados.rados.0.smithi195.stdout:227: writing smithi19530025-37 from 2244608 to 2850816 tid 2 2024-09-16T02:14:55.045 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:227: writing smithi19530025-37 from 2850816 to 3407872 tid 3 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:228: snap_create 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: finishing write tid 6 to smithi19530025-46 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: oid 46 updating version 0 to 54 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: oid 46 version 54 is already newer than 53 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: oid 46 version 54 is already newer than 51 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: oid 46 updating version 54 to 55 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: oid 46 version 55 is already newer than 52 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 55 (ObjNum 98 snap 16 seq_num 98) dirty exists 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:220: left oid 46 (ObjNum 98 snap 16 seq_num 98) 2024-09-16T02:14:55.046 INFO:tasks.rados.rados.0.smithi195.stdout:221: expect (ObjNum 96 snap 16 seq_num 96) 2024-09-16T02:14:55.047 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.085 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.099 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:14:55.136 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 2373348 2024-09-16T02:14:55.145 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:55.143+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:55.169 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:55.211 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:55.207+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:55.221 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.221 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.228 INFO:tasks.rados.rados.0.smithi195.stdout:224: finishing copy_from racing read to smithi19530025-8 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:224: finishing copy_from to smithi19530025-8 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 8 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:227: finishing write tid 1 to smithi19530025-37 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:227: finishing write tid 2 to smithi19530025-37 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:227: finishing write tid 3 to smithi19530025-37 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:226: finishing copy_from racing read to smithi19530025-47 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:226: finishing copy_from to smithi19530025-47 2024-09-16T02:14:55.229 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 47 v 30 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-16T02:14:55.236 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:14:55.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:14:55.321 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:55.319+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:55.400 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.401 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.413 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:55.406+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:55.415 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:14:55.488 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:55.488 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:55.514 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:55.514+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:55.517 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:55.518 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:55.546 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:55.547 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:55.564 INFO:tasks.rados.rados.0.smithi195.stdout:227: finishing write tid 4 to smithi19530025-37 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: finishing write tid 5 to smithi19530025-37 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: oid 37 updating version 0 to 61 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: oid 37 updating version 61 to 64 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: oid 37 version 64 is already newer than 63 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: oid 37 version 64 is already newer than 62 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 64 (ObjNum 99 snap 16 seq_num 99) dirty exists 2024-09-16T02:14:55.565 INFO:tasks.rados.rados.0.smithi195.stdout:227: left oid 37 (ObjNum 99 snap 16 seq_num 99) 2024-09-16T02:14:55.579 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:55.579 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:55.580 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.580 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.594 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:14:55.609 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:55.609 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:55.615 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:14:55.615+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:14:55.644 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:55.644 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:55.674 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:55.675 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:55.707 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:55.708 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:55.712 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.712 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.717 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:55.715+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:55.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:14:55.740 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:55.740 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:55.774 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:55.774 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:55.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:55.809 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:55.817 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:55.815+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:55.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:55.845 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:55.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:55.882 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:55.890 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:55.890 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:55.905 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:220: done (5 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:221: done (4 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:224: done (3 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:226: done (2 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:227: done (1 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:228: done (0 left) 2024-09-16T02:14:55.908 INFO:tasks.rados.rados.0.smithi195.stdout:229: read oid 37 snap -1 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:229: expect (ObjNum 99 snap 16 seq_num 99) 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:230: rollback oid 40 current snap is 17 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 40 to 11 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:231: read oid 7 snap -1 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:231: expect (ObjNum 89 snap 12 seq_num 89) 2024-09-16T02:14:55.909 INFO:tasks.rados.rados.0.smithi195.stdout:232: snap_create 2024-09-16T02:14:55.919 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:55.918+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:55.925 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:55.926 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:55.927 INFO:tasks.rados.rados.0.smithi195.stdout:230: finishing rollback tid 0 to smithi19530025-40 2024-09-16T02:14:55.932 INFO:tasks.rados.rados.0.smithi195.stdout:230: finishing rollback tid 1 to smithi19530025-40 2024-09-16T02:14:55.934 INFO:tasks.rados.rados.0.smithi195.stdout:230: finishing rollback tid 2 to smithi19530025-40 2024-09-16T02:14:55.934 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 40 v 53 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:14:55.936 INFO:tasks.rados.rados.0.smithi195.stdout:229: expect (ObjNum 99 snap 16 seq_num 99) 2024-09-16T02:14:55.937 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:14:55.964 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:55.965 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:55.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:55.958+0000 7fd6cad9d640 0 -- 172.21.15.3:0/1892556602 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7fd6cc2277e0 msgr2=0x7fd6cc229bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:14:55.991 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1157777 2024-09-16T02:14:56.019 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:56.015+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:56.028 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1560470 2024-09-16T02:14:56.053 INFO:tasks.rados.rados.0.smithi195.stdout:231: expect (ObjNum 89 snap 12 seq_num 89) 2024-09-16T02:14:56.061 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1424362 2024-09-16T02:14:56.065 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 364251 2024-09-16T02:14:56.078 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:14:56.078 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=114, reqid=client.4598.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.783018116000001, 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014358203, 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:14:56.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294123+0000", 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157742 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294233+0000", 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109842 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294256+0000", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3779000000000001e-05 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294602+0000", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000345817 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294625+0000", 2024-09-16T02:14:56.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308228+0000", 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013602725 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308323+0000", 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4631999999999998e-05 2024-09-16T02:14:56.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=115, reqid=client.4598.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.779242244999999, 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010657378, 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.084 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297742+0000", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297835+0000", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2315e-05 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298241+0000", 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040658699999999997 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298273+0000", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1566000000000002e-05 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298698+0000", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042513000000000002 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298724+0000", 2024-09-16T02:14:56.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5681999999999999e-05 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308331+0000", 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096073479999999999 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308398+0000", 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6749999999999996e-05 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.088 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.779081850000001, 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013936119, 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:56.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7299999999999996e-07 2024-09-16T02:14:56.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298091+0000", 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000189612 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298361+0000", 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026983699999999998 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298377+0000", 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6619000000000001e-05 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298580+0000", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020297699999999999 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298602+0000", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1823000000000001e-05 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311758+0000", 2024-09-16T02:14:56.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013155635000000001 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311837+0000", 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9042999999999995e-05 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.094 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.778884741999999, 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013796841000000001, 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298099+0000", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2999999999999999e-07 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298132+0000", 2024-09-16T02:14:56.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3460999999999997e-05 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298437+0000", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000305149 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298455+0000", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8099000000000001e-05 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299840+0000", 2024-09-16T02:14:56.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001384826 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5151999999999999e-05 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311844+0000", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011978284000000001 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311895+0000", 2024-09-16T02:14:56.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1039999999999999e-05 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.5s3 35/32 ECSubWrite(tid=2, reqid=client.4598.0:13, at_version=35'3, trim_to=0'0, roll_forward_to=34'1))", 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.778097475999999, 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011300899, 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298887+0000", 2024-09-16T02:14:56.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5400000000000001e-06 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298992+0000", 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105472 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299113+0000", 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012086099999999999 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299135+0000", 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1679e-05 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299294+0000", 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015895199999999999 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299322+0000", 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8246e-05 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310061+0000", 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010739165 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310186+0000", 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012498399999999999 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.775806533000001, 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010788813, 2024-09-16T02:14:56.104 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301178+0000", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.77e-06 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301420+0000", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024188200000000001 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301595+0000", 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017550199999999999 2024-09-16T02:14:56.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301613+0000", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7391999999999999e-05 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302202+0000", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000588903 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302225+0000", 2024-09-16T02:14:56.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311905+0000", 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096801509999999997 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311965+0000", 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9593999999999999e-05 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.775556072000001, 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013049734, 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:56.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6052000000000001e-05 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000240423 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301716+0000", 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2289e-05 2024-09-16T02:14:56.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305459+0000", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037425710000000001 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305488+0000", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8756e-05 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314378+0000", 2024-09-16T02:14:56.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088900869999999996 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314476+0000", 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8669e-05 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.775169936000001, 2024-09-16T02:14:56.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014210883000000001, 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:14:56.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301814+0000", 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4130000000000001e-06 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302022+0000", 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000207844 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302120+0000", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7864999999999999e-05 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302140+0000", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0156000000000001e-05 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310588+0000", 2024-09-16T02:14:56.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084482120000000001 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310612+0000", 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3879e-05 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315921+0000", 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053087079999999997 2024-09-16T02:14:56.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316024+0000", 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102806 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.774158557, 2024-09-16T02:14:56.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013263980999999999, 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302826+0000", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9240000000000001e-06 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302901+0000", 2024-09-16T02:14:56.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5172999999999993e-05 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304024+0000", 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001123181 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304072+0000", 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8127999999999997e-05 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312045+0000", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079722669999999999 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312071+0000", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6291999999999998e-05 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316033+0000", 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039623380000000001 2024-09-16T02:14:56.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316088+0000", 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4678000000000001e-05 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:49 3.cs0 3:3203a6be:::smithi19530025-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.761210524999999, 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016954778, 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.123 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315781+0000", 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8699999999999998e-06 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316702+0000", 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092113000000000004 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316734+0000", 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2375000000000001e-05 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316756+0000", 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1843000000000001e-05 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317070+0000", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031399899999999997 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317385+0000", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031517099999999999 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.320926+0000", 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035407450000000001 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332340+0000", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011414260000000001 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332532+0000", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191137 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332727+0000", 2024-09-16T02:14:56.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000195248 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:50 3.cs0 3:3203a6be:::smithi19530025-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.760169742999999, 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017275975999999998, 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:56.129 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316815+0000", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0899999999999999e-06 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316826+0000", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1242e-05 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316851+0000", 2024-09-16T02:14:56.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4969000000000001e-05 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316868+0000", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6535999999999999e-05 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324308+0000", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074406289999999998 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324542+0000", 2024-09-16T02:14:56.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023414200000000001 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.325642+0000", 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001099182 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332739+0000", 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070972090000000002 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:14:56.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.333048+0000", 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030949200000000002 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334089+0000", 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010404850000000001 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:51 3.cs0 3:3203a6be:::smithi19530025-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.760084247000002, 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019761212, 2024-09-16T02:14:56.134 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 51 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316900+0000", 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.477e-06 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316907+0000", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8889999999999996e-06 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316927+0000", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0591999999999999e-05 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316943+0000", 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5756999999999998e-05 2024-09-16T02:14:56.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327066+0000", 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010123037 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327269+0000", 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309000000000001 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334254+0000", 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069844 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334498+0000", 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024477499999999997 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336660+0000", 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021611949999999999 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/32 ECSubWrite(tid=224, reqid=osd.13.0:224, at_version=51'45, trim_to=0'0, roll_forward_to=49'43))", 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.662067725, 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24539247, 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2000000000000001e-07 2024-09-16T02:14:56.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414963+0000", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8265e-05 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658566+0000", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24360277699999999 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658581+0000", 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4253000000000001e-05 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658749+0000", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000168691 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658768+0000", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8332999999999999e-05 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660236+0000", 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014682129999999999 2024-09-16T02:14:56.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660307+0000", 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1618000000000004e-05 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.1s1 62/61 ECSubWriteReply(tid=22, last_complete=62'113, committed=1, applied=1))", 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.0454190729999997, 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66094662900000001, 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2300000000000002e-07 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031567+0000", 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3830000000000001e-06 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690362+0000", 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.65879452900000002 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690376+0000", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.397e-05 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690498+0000", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012224 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.692510+0000", 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020120839999999999 2024-09-16T02:14:56.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:525 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3854006549999998, 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014305174, 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.149 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 525 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:14:56.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691585+0000", 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.667e-06 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691596+0000", 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1837e-05 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691625+0000", 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8714999999999999e-05 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691640+0000", 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5027e-05 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.693656+0000", 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020154190000000001 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.694058+0000", 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040255 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.153 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.705887+0000", 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011828959 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:526 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3853146970000001, 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018909656, 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.154 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 526 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691669+0000", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.19e-06 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691676+0000", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.277e-06 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691700+0000", 2024-09-16T02:14:56.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3842e-05 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691718+0000", 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.823e-05 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695237+0000", 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035182619999999999 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695450+0000", 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021319899999999999 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.710578+0000", 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015127656 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:527 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3852393020000004, 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023766461999999999, 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 527 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691744+0000", 2024-09-16T02:14:56.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2299999999999999e-07 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691749+0000", 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0370000000000002e-06 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691763+0000", 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.342e-05 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691770+0000", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5320000000000001e-06 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.697667+0000", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058967209999999997 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.698047+0000", 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000379768 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715510+0000", 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017463060999999998 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpReadReply(3.1s1 62/61 ECSubReadReply(tid=46, attrs_read=0))", 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.700121+0000", 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3768612889999998, 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01053171, 2024-09-16T02:14:56.163 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.700121+0000", 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.700121+0000", 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.700122+0000", 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.083e-06 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.701531+0000", 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014082020000000001 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702235+0000", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000704559 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702243+0000", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7829999999999999e-06 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.706053+0000", 2024-09-16T02:14:56.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0038105159999999999 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.710653+0000", 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0045995669999999997 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpReadReply(3.1s1 62/61 ECSubReadReply(tid=47, attrs_read=0))", 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.3746492410000002, 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013250526, 2024-09-16T02:14:56.167 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702334+0000", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.074e-06 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702822+0000", 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048715800000000001 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702872+0000", 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0565000000000001e-05 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702881+0000", 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.49e-06 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.711121+0000", 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082399199999999995 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715584+0000", 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044633190000000003 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1s1 62/61 [PushReplyOp(3:830a71a8:::smithi19530025-14:head)])", 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:14:56.171 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.372769087, 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19260793200000001, 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:14:56.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1399999999999997e-07 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704220+0000", 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1759999999999998e-06 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.173 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894917+0000", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19069640900000001 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894932+0000", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5690000000000001e-05 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.895110+0000", 2024-09-16T02:14:56.174 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000177546 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.896821+0000", 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017116970000000001 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:14:56.175 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:14:56.176 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 678692 2024-09-16T02:14:56.177 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:14:56.178 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:56.174+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:56.203 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:56.203 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:56.278 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:56.274+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:56.305 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:56.305 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:56.305 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:56.306 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:56.320 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:14:56.379 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:56.378+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:56.441 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:56.442 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:56.442 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:56.442 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:56.442 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:56.454 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:14:56.480 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:56.475+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 29.241865805, 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.334730205, 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.581 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.582 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335879+0000", 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335907+0000", 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7291999999999999e-05 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669030+0000", 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33312349400000002 2024-09-16T02:14:56.583 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669045+0000", 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4943000000000001e-05 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669245+0000", 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019942899999999999 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.584 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669263+0000", 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8445999999999999e-05 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670533+0000", 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001270024 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670602+0000", 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8406999999999997e-05 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.586 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.242892998999999, 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75881597999999995, 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.587 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334845+0000", 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0900000000000001e-07 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334886+0000", 2024-09-16T02:14:56.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1254999999999997e-05 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091945+0000", 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75705888700000001 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091960+0000", 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4686e-05 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092157+0000", 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019717000000000001 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092176+0000", 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9020000000000001e-05 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093578+0000", 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001401769 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093660+0000", 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2484000000000005e-05 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.593 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 45/32 ECSubWrite(tid=22, reqid=osd.6.0:21, at_version=0'0, trim_to=0'0, roll_forward_to=45'16))", 2024-09-16T02:14:56.593 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:56.593 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.226401414000001, 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061315420000000002, 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351336+0000", 2024-09-16T02:14:56.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351337+0000", 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.773e-06 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.596 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351366+0000", 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8408999999999999e-05 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351446+0000", 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0513999999999995e-05 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351467+0000", 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0913000000000001e-05 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352256+0000", 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00078906799999999997 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352279+0000", 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2186000000000001e-05 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357119+0000", 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048408469999999997 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357467+0000", 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034783200000000001 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.225869900999999, 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017058792, 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351868+0000", 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351917+0000", 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352051+0000", 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013457599999999999 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352073+0000", 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1489e-05 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352231+0000", 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158455 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352260+0000", 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8472000000000001e-05 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368854+0000", 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016594647000000001 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368926+0000", 2024-09-16T02:14:56.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1534999999999999e-05 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs5 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.219834768999998, 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0088459079999999995, 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.605 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357902+0000", 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357905+0000", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3599999999999999e-06 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357997+0000", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2616999999999998e-05 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358121+0000", 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012370600000000001 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.607 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358231+0000", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109963 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358421+0000", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019028099999999999 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358634+0000", 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021316999999999999 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366661+0000", 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080266509999999992 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.366748+0000", 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7159999999999999e-05 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.218997652999999, 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092364969999999998, 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358739+0000", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358741+0000", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9580000000000001e-06 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358753+0000", 2024-09-16T02:14:56.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1242e-05 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358833+0000", 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9919999999999994e-05 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358861+0000", 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8904999999999999e-05 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367924+0000", 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090629869999999998 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367976+0000", 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1484999999999999e-05 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.613 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.218168008999999, 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085149430000000005, 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359569+0000", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359571+0000", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.68e-06 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359582+0000", 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1399999999999999e-05 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359660+0000", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7704e-05 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359694+0000", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4062e-05 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368042+0000", 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083476850000000005 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368084+0000", 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2412e-05 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=33, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.218086833000001, 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086774060000000004, 2024-09-16T02:14:56.617 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359650+0000", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359652+0000", 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.114e-06 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359664+0000", 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1617000000000001e-05 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359750+0000", 2024-09-16T02:14:56.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6180000000000005e-05 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.359794+0000", 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3962999999999999e-05 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368284+0000", 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084900930000000006 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368328+0000", 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3439000000000002e-05 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.217615656, 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079952310000000002, 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.621 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360121+0000", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360122+0000", 2024-09-16T02:14:56.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7699999999999992e-07 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360169+0000", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6087000000000001e-05 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360275+0000", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106805 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360296+0000", 2024-09-16T02:14:56.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0744999999999999e-05 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360496+0000", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020018400000000001 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.360529+0000", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2329000000000002e-05 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368025+0000", 2024-09-16T02:14:56.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074963929999999996 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368117+0000", 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1711000000000003e-05 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.215327388999999, 2024-09-16T02:14:56.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079995459999999997, 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362410+0000", 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362411+0000", 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.53e-06 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362463+0000", 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1981999999999998e-05 2024-09-16T02:14:56.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362640+0000", 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000177047 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.362687+0000", 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6349e-05 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365835+0000", 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031487730000000001 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365859+0000", 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3612999999999999e-05 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.370339+0000", 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044803380000000004 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.370409+0000", 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9913999999999995e-05 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.630 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.209952377, 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0076441920000000002, 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367785+0000", 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.63e-07 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.367837+0000", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2117999999999998e-05 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372037+0000", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042000819999999999 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372060+0000", 2024-09-16T02:14:56.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2782999999999999e-05 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372212+0000", 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000152206 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.372234+0000", 2024-09-16T02:14:56.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1475999999999999e-05 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375366+0000", 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031322160000000002 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375429+0000", 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2848000000000005e-05 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.635 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=35, last_complete=45'32, committed=1, applied=1))", 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.195955876999999, 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65850650099999997, 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.636 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381783+0000", 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.857e-06 2024-09-16T02:14:56.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381795+0000", 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1466e-05 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040081+0000", 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65828677099999999 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040095+0000", 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3686000000000001e-05 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040215+0000", 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011949 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040288+0000", 2024-09-16T02:14:56.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3231000000000004e-05 2024-09-16T02:14:56.644 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.645 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.645 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=24, reqid=client.4598.0:254, at_version=45'15, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.185257311000001, 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0063935490000000001, 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.646 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392480+0000", 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392481+0000", 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.282e-06 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392591+0000", 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109607 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397038+0000", 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044475239999999996 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397053+0000", 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4962e-05 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397183+0000", 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012977600000000001 2024-09-16T02:14:56.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397207+0000", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3564e-05 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.398820+0000", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001613094 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.398873+0000", 2024-09-16T02:14:56.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3739999999999997e-05 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:56.652 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.180749784, 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27792197099999999, 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396988+0000", 2024-09-16T02:14:56.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759999999999999e-06 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397010+0000", 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1525000000000001e-05 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673630+0000", 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27662027 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673655+0000", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5003000000000001e-05 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673857+0000", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020175299999999999 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673875+0000", 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:14:56.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674832+0000", 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095631999999999998 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674909+0000", 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7741e-05 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.657 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:487 3.7s0 3:ed091bc8:::smithi19530025-23:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc d=[d,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.645123783000001, 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019918874, 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:56.658 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 487 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932617+0000", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.309e-06 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932695+0000", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8548999999999996e-05 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932736+0000", 2024-09-16T02:14:56.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0349000000000003e-05 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932757+0000", 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1903e-05 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932928+0000", 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017018600000000001 2024-09-16T02:14:56.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932950+0000", 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2395000000000002e-05 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934568+0000", 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00161812 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934773+0000", 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000204995 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936578+0000", 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001804709 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.940066+0000", 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034883869999999999 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952401+0000", 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012334539 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952532+0000", 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131433 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.636722006999999, 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013412999, 2024-09-16T02:14:56.665 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941016+0000", 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6000000000000004e-07 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941065+0000", 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9462000000000002e-05 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941113+0000", 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941128+0000", 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4348999999999999e-05 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941254+0000", 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012600400000000001 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941277+0000", 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3482999999999999e-05 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954333+0000", 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013055430999999999 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954428+0000", 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5468000000000001e-05 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 57/32 ECSubWrite(tid=118, reqid=osd.11.0:117, at_version=0'0, trim_to=0'0, roll_forward_to=57'104))", 2024-09-16T02:14:56.670 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.947707+0000", 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.630030546, 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080366420000000001, 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947707+0000", 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947707+0000", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947708+0000", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.88e-06 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947771+0000", 2024-09-16T02:14:56.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-05 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947824+0000", 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2784999999999997e-05 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947834+0000", 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0219e-05 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947903+0000", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8888999999999997e-05 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.947913+0000", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.859e-06 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.955657+0000", 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077433060000000001 2024-09-16T02:14:56.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.955743+0000", 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6704000000000002e-05 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=50, reqid=client.4598.0:494, at_version=58'42, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.6187910609999996, 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010360967, 2024-09-16T02:14:56.675 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958947+0000", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4300000000000003e-07 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959048+0000", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101671 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959143+0000", 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5120999999999999e-05 2024-09-16T02:14:56.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959174+0000", 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0803000000000003e-05 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959376+0000", 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202105 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959404+0000", 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7141999999999999e-05 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969241+0000", 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098377399999999993 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969307+0000", 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5741999999999995e-05 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.680 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.6120588419999997, 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011275655000000001, 2024-09-16T02:14:56.681 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965681+0000", 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2199999999999999e-06 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965731+0000", 2024-09-16T02:14:56.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0429000000000001e-05 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965878+0000", 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014756 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965904+0000", 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5686000000000001e-05 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967710+0000", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001805652 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.968220+0000", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050980399999999999 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976848+0000", 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086280009999999997 2024-09-16T02:14:56.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976954+0000", 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106303 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.687 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=52, reqid=client.4598.0:496, at_version=58'44, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:56.687 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:14:56.687 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.6119947999999997, 2024-09-16T02:14:56.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011274895, 2024-09-16T02:14:56.687 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:56.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:14:56.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:56.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.689 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:56.689 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965743+0000", 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1069999999999999e-06 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965783+0000", 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9165999999999997e-05 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967647+0000", 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018640340000000001 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967671+0000", 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4712e-05 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971326+0000", 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036543130000000002 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971353+0000", 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7376e-05 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:56.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976963+0000", 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056098989999999998 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.977017+0000", 2024-09-16T02:14:56.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4287999999999999e-05 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:56.694 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:56.695 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:14:56.578+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:14:56.696 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:14:56.697 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:14:56.694+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:14:56.797 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:56.795+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:56.825 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:56.825 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:56.825 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:56.826 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:56.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:14:56.897 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:56.894+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:56.914 INFO:tasks.rados.rados.0.smithi195.stdout:229: done (3 left) 2024-09-16T02:14:56.914 INFO:tasks.rados.rados.0.smithi195.stdout:230: done (2 left) 2024-09-16T02:14:56.914 INFO:tasks.rados.rados.0.smithi195.stdout:231: done (1 left) 2024-09-16T02:14:56.914 INFO:tasks.rados.rados.0.smithi195.stdout:232: done (0 left) 2024-09-16T02:14:56.914 INFO:tasks.rados.rados.0.smithi195.stdout:233: snap_create 2024-09-16T02:14:56.999 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:56.998+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:57.012 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:57.012 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:57.012 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:57.012 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:57.012 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:57.026 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:14:57.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:14:57.099 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:57.095+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:57.194 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:476 3.bs0 3:d814c48f:::smithi19530025-7:head [append 0~737280] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:44.986147+0000", 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.205224821, 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014754187, 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.195 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 476 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.986147+0000", 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.196 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.986147+0000", 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.986151+0000", 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7639999999999999e-06 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.197 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.987351+0000", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012001030000000001 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.987396+0000", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5339000000000001e-05 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.987421+0000", 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4700000000000001e-05 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.198 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.987616+0000", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019571899999999999 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.987632+0000", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5446000000000001e-05 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.991368+0000", 2024-09-16T02:14:57.199 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037364709999999999 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.991434+0000", 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5685000000000006e-05 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.992904+0000", 2024-09-16T02:14:57.200 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001469698 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.997670+0000", 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004766157 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.000808+0000", 2024-09-16T02:14:57.201 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0031383399999999999 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.000901+0000", 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2764999999999997e-05 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:479 3.bs0 3:d814c48f:::smithi19530025-7:head [setxattr _header (56) in=7b] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:44.996295+0000", 2024-09-16T02:14:57.202 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.195076405, 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098257499999999994, 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 479 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.203 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996295+0000", 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996295+0000", 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.204 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996296+0000", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1909999999999999e-06 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996305+0000", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2500000000000006e-06 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996319+0000", 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4567e-05 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.205 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996330+0000", 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0553e-05 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.206 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.999137+0000", 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028076939999999999 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.999231+0000", 2024-09-16T02:14:57.207 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3177000000000006e-05 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.999696+0000", 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00046496500000000002 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001300+0000", 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016045390000000001 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.208 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.005997+0000", 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046969519999999999 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006121+0000", 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123862 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.209 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:480 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:44.996346+0000", 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.195025525, 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014599279, 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 480 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.210 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996346+0000", 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996346+0000", 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.211 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996347+0000", 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5000000000000001e-07 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996353+0000", 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4470000000000003e-06 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996366+0000", 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2547e-05 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.212 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.996375+0000", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0960000000000007e-06 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:44.999934+0000", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035584229999999998 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.000018+0000", 2024-09-16T02:14:57.213 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4377999999999997e-05 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006168+0000", 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0061503360000000002 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006251+0000", 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3096000000000005e-05 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.214 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.010945+0000", 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046940059999999997 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:481 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [append 0~753664] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.001558+0000", 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.189813426000001, 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010733548000000001, 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.215 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 481 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001558+0000", 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.216 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001558+0000", 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001560+0000", 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.437e-06 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.217 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001913+0000", 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000353433 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001931+0000", 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7711e-05 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001939+0000", 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0080000000000002e-06 2024-09-16T02:14:57.218 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.001988+0000", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8785000000000001e-05 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.002079+0000", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1002999999999996e-05 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.003180+0000", 2024-09-16T02:14:57.219 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011015669999999999 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.004689+0000", 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015088090000000001 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012179+0000", 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074899210000000001 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.220 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012292+0000", 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000112874 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:484 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [setxattr _header (57) in=7b] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:14:57.221 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.184636552000001, 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62580566599999998, 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 484 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:14:57.222 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006736+0000", 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2500000000000004e-07 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.223 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006740+0000", 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6460000000000001e-06 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006749+0000", 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0390000000000005e-06 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006754+0000", 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1939999999999997e-06 2024-09-16T02:14:57.224 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011015+0000", 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612830000000003 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.225 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011096+0000", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0894000000000002e-05 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011534+0000", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043832300000000001 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012404+0000", 2024-09-16T02:14:57.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086940800000000001 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632372+0000", 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.619968348 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632541+0000", 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016870599999999999 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.227 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:485 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.184610179, 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.628869487, 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.228 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 485 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.229 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006762+0000", 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4499999999999997e-07 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006765+0000", 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9270000000000001e-06 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006771+0000", 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3679999999999998e-06 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006777+0000", 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2959999999999998e-06 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012433+0000", 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056561729999999996 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012499+0000", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6303999999999994e-05 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633369+0000", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62087044899999999 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633608+0000", 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023843099999999999 2024-09-16T02:14:57.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.635631+0000", 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020229940000000002 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 56/32 ECSubWriteReply(tid=59, last_complete=56'17, committed=1, applied=1))", 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.178643658, 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62052908399999995, 2024-09-16T02:14:57.234 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:14:57.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012730+0000", 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.694e-06 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012741+0000", 2024-09-16T02:14:57.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1343e-05 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632045+0000", 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61930391299999998 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632066+0000", 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0699999999999998e-05 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632237+0000", 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017166600000000001 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633257+0000", 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010197680000000001 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s3 57/32 ECSubWrite(tid=59, reqid=client.4598.0:487, at_version=57'55, trim_to=0'0, roll_forward_to=54'53))", 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.936733+0000", 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.254638379999999, 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015094955, 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936733+0000", 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936733+0000", 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936734+0000", 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9400000000000005e-07 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936960+0000", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022612700000000001 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.937079+0000", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118997 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.937105+0000", 2024-09-16T02:14:57.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6250000000000001e-05 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.937270+0000", 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000165191 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.937291+0000", 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0635000000000001e-05 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.951780+0000", 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014488829 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.951828+0000", 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8232000000000002e-05 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:14:57.243 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.942089+0000", 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.249282482, 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098448540000000001, 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942089+0000", 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942089+0000", 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942090+0000", 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942151+0000", 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1672000000000002e-05 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942282+0000", 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000130813 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.942304+0000", 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1869e-05 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.944895+0000", 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0025913799999999999 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.944911+0000", 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5685999999999999e-05 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.951902+0000", 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0069904260000000001 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.951934+0000", 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2378000000000001e-05 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.248 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:497 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.235152739, 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028762988, 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.249 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 497 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956222+0000", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6200000000000001e-06 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956239+0000", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6458999999999999e-05 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956280+0000", 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0942999999999997e-05 2024-09-16T02:14:57.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956303+0000", 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2742999999999999e-05 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956495+0000", 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000192762 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:57.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956517+0000", 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1375000000000001e-05 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962337+0000", 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058203600000000001 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962579+0000", 2024-09-16T02:14:57.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024195400000000001 2024-09-16T02:14:57.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.984982+0000", 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022402772000000001 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:498 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.257 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.235021216, 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030739767000000001, 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 498 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956352+0000", 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.75e-06 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956362+0000", 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6900000000000004e-06 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956408+0000", 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5828999999999998e-05 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956426+0000", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8216e-05 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956684+0000", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025823399999999998 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956702+0000", 2024-09-16T02:14:57.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963967+0000", 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072649940000000003 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.966462+0000", 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024953639999999999 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.987090+0000", 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020627774000000001 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:499 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.234916255, 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034753205000000002, 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.263 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 499 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956457+0000", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5969999999999999e-06 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956469+0000", 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1929999999999999e-05 2024-09-16T02:14:57.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956493+0000", 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3849000000000002e-05 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956509+0000", 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.266 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956872+0000", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000363424 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956891+0000", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8913e-05 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967274+0000", 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010382466 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967524+0000", 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025058799999999999 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.991209+0000", 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023684105 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.268 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:500 3.4s0 3:23d2836d:::smithi19530025-21:head [rmxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt in=40b,rmxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ in=40b,rmxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q in=40b,rmxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd in=35b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.234792594, 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030187572999999999, 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 500 2024-09-16T02:14:57.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:14:57.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956581+0000", 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6959999999999999e-06 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956591+0000", 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.025e-05 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956614+0000", 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3068999999999999e-05 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956632+0000", 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8199999999999999e-05 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956786+0000", 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015396999999999999 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956813+0000", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6872e-05 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963384+0000", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065708399999999997 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963658+0000", 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027420299999999999 2024-09-16T02:14:57.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965812+0000", 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021535809999999999 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.981888+0000", 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016076136000000001 2024-09-16T02:14:57.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986582+0000", 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046937309999999996 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986767+0000", 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018502500000000001 2024-09-16T02:14:57.275 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 58/32 ECSubWrite(tid=50, reqid=client.4598.0:494, at_version=58'42, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.960128+0000", 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.231243979, 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011079309000000001, 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960128+0000", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960128+0000", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960128+0000", 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7700000000000004e-07 2024-09-16T02:14:57.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960184+0000", 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5328999999999998e-05 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960387+0000", 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000203644 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960416+0000", 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8875999999999999e-05 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960582+0000", 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000165663 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.960609+0000", 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6988999999999999e-05 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971124+0000", 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010515656999999999 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971207+0000", 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2573999999999999e-05 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds5 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.963214+0000", 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.228157993, 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010182583, 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.281 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963214+0000", 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963214+0000", 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963215+0000", 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.787e-06 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963262+0000", 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6925999999999998e-05 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963355+0000", 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2357999999999999e-05 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963376+0000", 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1178999999999998e-05 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969894+0000", 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065179319999999997 2024-09-16T02:14:57.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969917+0000", 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3620000000000001e-05 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.973311+0000", 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033936589999999998 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.973396+0000", 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5122000000000003e-05 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.286 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 61/60 ECSubWriteReply(tid=78, last_complete=61'24, committed=1, applied=1))", 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.00242348, 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45998103899999998, 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188949+0000", 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5700000000000002e-07 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188953+0000", 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5129999999999998e-06 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647595+0000", 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.458642198 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647616+0000", 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0886e-05 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647777+0000", 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016024999999999999 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.648929+0000", 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011525350000000001 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:590 3.bs0 3:d814c48f:::smithi19530025-7:head [read 364251~1257765,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:55.911309+0000", 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.280062185, 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014176882999999999, 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.291 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 590 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911309+0000", 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911309+0000", 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911312+0000", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4870000000000001e-06 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911327+0000", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4859e-05 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911351+0000", 2024-09-16T02:14:57.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3983000000000001e-05 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911368+0000", 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.715e-05 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911486+0000", 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000117767 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911727+0000", 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024184200000000001 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.925486+0000", 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013758795000000001 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.295 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:591 3.bs0 3:d814c48f:::smithi19530025-7:head [read 678692~943324,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:55.911444+0000", 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.279927528, 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019327939999999998, 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 591 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911444+0000", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911444+0000", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911446+0000", 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6640000000000001e-06 2024-09-16T02:14:57.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911455+0000", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7259999999999997e-06 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911481+0000", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5119000000000001e-05 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.911601+0000", 2024-09-16T02:14:57.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012063200000000001 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.912848+0000", 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012468239999999999 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.913087+0000", 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023917099999999999 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.930772+0000", 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017684803999999998 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpReadReply(3.bs0 66/32 ECSubReadReply(tid=89, attrs_read=0))", 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:55.915616+0000", 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.2757557310000001, 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099245210000000004, 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.915616+0000", 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.915616+0000", 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.915618+0000", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.827e-06 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.920288+0000", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004670507 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.921697+0000", 2024-09-16T02:14:57.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0014086299999999999 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.921717+0000", 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9796000000000002e-05 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.923949+0000", 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0022327480000000001 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.925540+0000", 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015910130000000001 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpReadReply(3.bs0 66/32 ECSubReadReply(tid=90, attrs_read=0))", 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:55.920301+0000", 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.271070438, 2024-09-16T02:14:57.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010606289, 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.920301+0000", 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.920301+0000", 2024-09-16T02:14:57.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.920302+0000", 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3030000000000001e-06 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.920809+0000", 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050624299999999997 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.921782+0000", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097298999999999997 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.921795+0000", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.289e-05 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.929630+0000", 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078352530000000004 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:55.930907+0000", 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00127761 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.308 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:57.310 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:14:57.310 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:57.302+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:57.410 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:57.411+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:57.441 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:57.441 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:57.441 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:57.457 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:14:57.512 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:57.507+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:57.575 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:57.575 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:57.575 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:57.576 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:57.576 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:57.590 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:14:57.612 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:57.611+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:57.713 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:57.711+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:31 3.1s0 3:89ce0a26:::smithi19530025-11:head [append 0~180224,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.715 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.421101788000001, 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022356674999999999, 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 31 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290656+0000", 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3159999999999999e-06 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290984+0000", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032549899999999998 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.290996+0000", 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2635e-05 2024-09-16T02:14:57.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291004+0000", 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5440000000000001e-06 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291070+0000", 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6177e-05 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291178+0000", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107766 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295206+0000", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040278029999999999 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299290+0000", 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0040844139999999998 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312901+0000", 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013611165999999999 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313013+0000", 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011135499999999999 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:32 3.1s0 3:89ce0a26:::smithi19530025-11:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420726457000001, 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022289759999999999, 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.722 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 32 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291031+0000", 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291032+0000", 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2600000000000001e-07 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291036+0000", 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2760000000000002e-06 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291044+0000", 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7190000000000001e-06 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291050+0000", 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.5779999999999998e-06 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295781+0000", 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0047316909999999997 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.295883+0000", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010134 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296411+0000", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052795499999999998 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299292+0000", 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028809719999999999 2024-09-16T02:14:57.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313180+0000", 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013887932 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313321+0000", 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000141371 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.727 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:33 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420694232999999, 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024514080000000001, 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.728 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 33 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291064+0000", 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291067+0000", 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7199999999999998e-06 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291074+0000", 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4680000000000003e-06 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291080+0000", 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.964e-06 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055391950000000002 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0838000000000001e-05 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313405+0000", 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016704404999999999 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313545+0000", 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014004999999999999 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315578+0000", 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020328199999999999 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:37 3.1s0 3:81c6c0a8:::smithi19530025-13:head [append 0~212992,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420665786000001, 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023064575, 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.734 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 37 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291092+0000", 2024-09-16T02:14:57.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291093+0000", 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4399999999999999e-07 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291317+0000", 2024-09-16T02:14:57.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022428500000000001 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291335+0000", 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7980999999999999e-05 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291347+0000", 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1708e-05 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296736+0000", 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053889660000000002 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296859+0000", 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000123786 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297520+0000", 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00066093900000000004 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302005+0000", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044842199999999997 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314113+0000", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012108659000000001 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314156+0000", 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.3186999999999998e-05 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.739 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:38 3.1s0 3:81c6c0a8:::smithi19530025-13:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420396300999997, 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023044124999999999, 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.740 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 38 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291361+0000", 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291362+0000", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0100000000000001e-06 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291368+0000", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6879999999999996e-06 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291381+0000", 2024-09-16T02:14:57.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3369e-05 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291404+0000", 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2164e-05 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298518+0000", 2024-09-16T02:14:57.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007114029 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.298598+0000", 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0813e-05 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299017+0000", 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041827500000000002 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302007+0000", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029904630000000001 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314313+0000", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012305768999999999 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314405+0000", 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2545000000000002e-05 2024-09-16T02:14:57.745 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:39 3.1s0 3:81c6c0a8:::smithi19530025-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420324368000003, 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024374809000000001, 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.746 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 39 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291434+0000", 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291440+0000", 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4960000000000004e-06 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291456+0000", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025e-05 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291468+0000", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299421+0000", 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079526309999999999 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299510+0000", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9506000000000007e-05 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314495+0000", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014984548 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314572+0000", 2024-09-16T02:14:57.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6854999999999995e-05 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315808+0000", 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012362639999999999 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:40 3.1s0 3:830a71a8:::smithi19530025-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:57.751 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.420272775999997, 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024497693000000001, 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 40 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291486+0000", 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8999999999999995e-07 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291773+0000", 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028741999999999998 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291795+0000", 2024-09-16T02:14:57.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.181e-05 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291807+0000", 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1687e-05 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299547+0000", 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077406699999999998 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299647+0000", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9978999999999999e-05 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.300445+0000", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079766400000000001 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302061+0000", 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016158539999999999 2024-09-16T02:14:57.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315913+0000", 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013852573999999999 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315983+0000", 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9145000000000003e-05 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:41 3.1s0 3:830a71a8:::smithi19530025-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.419925442, 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024477925000000001, 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291833+0000", 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:14:57.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291839+0000", 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291855+0000", 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291865+0000", 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0037999999999999e-05 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301194+0000", 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093284930000000002 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301281+0000", 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6972999999999996e-05 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301794+0000", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051343099999999996 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.304892+0000", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030976950000000001 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316223+0000", 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011331071999999999 2024-09-16T02:14:57.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316310+0000", 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.742e-05 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:42 3.1s0 3:830a71a8:::smithi19530025-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.419879250000001, 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031481841000000003, 2024-09-16T02:14:57.763 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291880+0000", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2139999999999999e-06 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291883+0000", 2024-09-16T02:14:57.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291902+0000", 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8881000000000001e-05 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291912+0000", 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9680000000000001e-06 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302284+0000", 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371521999999999 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302334+0000", 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0047999999999999e-05 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318985+0000", 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016650888999999999 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.319121+0000", 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136336 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323360+0000", 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042391670000000003 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.768 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:247 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (56) in=7b] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 24.281196437999998, 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30233062700000002, 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 247 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430562+0000", 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1230000000000001e-06 2024-09-16T02:14:57.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430568+0000", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.519e-06 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430582+0000", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.366e-05 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430592+0000", 2024-09-16T02:14:57.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0482999999999999e-05 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433637+0000", 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003044705 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433734+0000", 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6916000000000005e-05 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434127+0000", 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039313800000000002 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435690+0000", 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015630640000000001 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732747+0000", 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29705759700000001 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732892+0000", 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014442200000000001 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:248 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 24.281121994999999, 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.304991454, 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.774 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 248 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430637+0000", 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3099999999999996e-07 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430641+0000", 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430654+0000", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638999999999999e-05 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430662+0000", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7370000000000006e-06 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434972+0000", 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043098019999999997 2024-09-16T02:14:57.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435055+0000", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3625e-05 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.733794+0000", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.298738691 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.734027+0000", 2024-09-16T02:14:57.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023328199999999999 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.735627+0000", 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015998379999999999 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:250 3.1s0 3:89ce0a26:::smithi19530025-11:head [copy-from ver 2] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:57.779 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.366697243000001, 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046806635999999999, 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 250 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345064+0000", 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.484e-06 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345079+0000", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4939999999999999e-05 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345114+0000", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5258e-05 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345131+0000", 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7356000000000001e-05 2024-09-16T02:14:57.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345312+0000", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018022200000000001 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345332+0000", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0509e-05 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:14:57.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011526471 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357011+0000", 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015221500000000001 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375994+0000", 2024-09-16T02:14:57.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018982974999999999 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380796+0000", 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048023010000000001 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391742+0000", 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010946141 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.785 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391867+0000", 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124764 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:251 3.1s0 3:89ce0a26:::smithi19530025-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.366579637000001, 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034174018, 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.786 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 251 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345179+0000", 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1799999999999999e-06 2024-09-16T02:14:57.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345186+0000", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5259999999999999e-06 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345202+0000", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6453999999999999e-05 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345213+0000", 2024-09-16T02:14:57.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1147e-05 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345556+0000", 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000342219 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345581+0000", 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5876000000000001e-05 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.790 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012465654 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358190+0000", 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014254599999999999 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379056+0000", 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020865887 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.791 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379212+0000", 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156675 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379352+0000", 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013985399999999999 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.792 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 45/32 ECSubWriteReply(tid=55, last_complete=45'49, committed=1, applied=1))", 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.318760620999999, 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.456257309, 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392998+0000", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2780000000000001e-06 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.393006+0000", 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1310000000000006e-06 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847561+0000", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45455489999999998 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847576+0000", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.453e-05 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847695+0000", 2024-09-16T02:14:57.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118799 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.849254+0000", 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015596710000000001 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:260 3.1s0 3:830a71a8:::smithi19530025-14:head [copy-from ver 2] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:57.796 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.354649054999999, 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02650228, 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 260 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357110+0000", 2024-09-16T02:14:57.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.243e-06 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357117+0000", 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9e-06 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357132+0000", 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5156e-05 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357141+0000", 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357434+0000", 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029301799999999998 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357457+0000", 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2821999999999999e-05 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365347+0000", 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007890088 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365650+0000", 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030261099999999999 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:57.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.379802+0000", 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014151762 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381743+0000", 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001941531 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383415+0000", 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016721349999999999 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383611+0000", 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019561399999999999 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:261 3.1s0 3:830a71a8:::smithi19530025-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.354568228000002, 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024496032000000001, 2024-09-16T02:14:57.803 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 261 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.804 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357191+0000", 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3799999999999999e-06 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357198+0000", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1740000000000003e-06 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357220+0000", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2322000000000001e-05 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357238+0000", 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7377e-05 2024-09-16T02:14:57.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357578+0000", 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034031900000000001 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357600+0000", 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1543e-05 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366106+0000", 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085068360000000003 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366241+0000", 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013419899999999999 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381121+0000", 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014880769 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381357+0000", 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023564300000000001 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381685+0000", 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032846999999999998 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:501 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 284996~1533628,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.75489522, 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023246494999999999, 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 501 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.810 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956866+0000", 2024-09-16T02:14:57.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6500000000000002e-06 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956875+0000", 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5469999999999997e-06 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956912+0000", 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6862e-05 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956935+0000", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3986e-05 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957148+0000", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021242200000000001 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957166+0000", 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7604999999999998e-05 2024-09-16T02:14:57.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969486+0000", 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012320806 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969825+0000", 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033867100000000002 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.980109+0000", 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010283946 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:502 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1658552~160072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.754763063, 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025237151999999999, 2024-09-16T02:14:57.815 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 502 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956996+0000", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.28e-06 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957005+0000", 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8459999999999997e-06 2024-09-16T02:14:57.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957029+0000", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4053000000000002e-05 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957045+0000", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6293000000000001e-05 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957336+0000", 2024-09-16T02:14:57.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029122600000000001 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957354+0000", 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7543000000000001e-05 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974088+0000", 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016734532999999999 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974356+0000", 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026803599999999998 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.982232+0000", 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078753420000000005 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:503 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1451430~367194,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:14:57.820 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.754637853, 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.32090356199999998, 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 503 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957122+0000", 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3470000000000001e-06 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957136+0000", 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4079e-05 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957168+0000", 2024-09-16T02:14:57.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2159000000000002e-05 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957189+0000", 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0367e-05 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957429+0000", 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000240561 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957446+0000", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6472000000000001e-05 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976521+0000", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019075392 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976775+0000", 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025338100000000001 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278023+0000", 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30124880399999998 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpReadReply(3.1s0 58/32 ECSubReadReply(tid=123, attrs_read=0))", 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:14:57.826 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 10.733283503999999, 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29957913899999999, 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:57.827 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978475+0000", 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2999999999999999e-07 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978563+0000", 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8211e-05 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:57.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277175+0000", 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29861169199999998 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277197+0000", 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1566999999999999e-05 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277332+0000", 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013569999999999999 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:57.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278053+0000", 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072103900000000003 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:57.830 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:57.832 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:14:57.833 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:14:57.830+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:14:57.909 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:14:57.916 INFO:tasks.rados.rados.0.smithi195.stdout:233: done (0 left) 2024-09-16T02:14:57.916 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 2 current snap is 19 2024-09-16T02:14:57.916 INFO:tasks.rados.rados.0.smithi195.stdout:234: seq_num 100 ranges {1245184=524288,1769472=753664,2523136=409600} 2024-09-16T02:14:57.929 INFO:tasks.rados.rados.0.smithi195.stdout:234: writing smithi19530025-2 from 1245184 to 1769472 tid 1 2024-09-16T02:14:57.933 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:57.931+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:57.935 INFO:tasks.rados.rados.0.smithi195.stdout:234: writing smithi19530025-2 from 1769472 to 2523136 tid 2 2024-09-16T02:14:57.938 INFO:tasks.rados.rados.0.smithi195.stdout:234: writing smithi19530025-2 from 2523136 to 2932736 tid 3 2024-09-16T02:14:57.938 INFO:tasks.rados.rados.0.smithi195.stdout:235: delete oid 43 current snap is 19 2024-09-16T02:14:57.943 INFO:tasks.rados.rados.0.smithi195.stdout:234: finishing write tid 1 to smithi19530025-2 2024-09-16T02:14:57.944 INFO:tasks.rados.rados.0.smithi195.stdout:234: finishing write tid 2 to smithi19530025-2 2024-09-16T02:14:57.944 INFO:tasks.rados.rados.0.smithi195.stdout:235: done (1 left) 2024-09-16T02:14:57.944 INFO:tasks.rados.rados.0.smithi195.stdout:236: delete oid 35 current snap is 19 2024-09-16T02:14:57.950 INFO:tasks.rados.rados.0.smithi195.stdout:236: done (1 left) 2024-09-16T02:14:57.951 INFO:tasks.rados.rados.0.smithi195.stdout:237: delete oid 38 current snap is 19 2024-09-16T02:14:57.952 INFO:tasks.rados.rados.0.smithi195.stdout:234: finishing write tid 3 to smithi19530025-2 2024-09-16T02:14:57.953 INFO:tasks.rados.rados.0.smithi195.stdout:234: finishing write tid 4 to smithi19530025-2 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: finishing write tid 5 to smithi19530025-2 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: oid 2 updating version 0 to 57 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: oid 2 updating version 57 to 58 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: oid 2 version 58 is already newer than 56 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: oid 2 version 58 is already newer than 55 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 2 v 58 (ObjNum 100 snap 19 seq_num 100) dirty exists 2024-09-16T02:14:57.957 INFO:tasks.rados.rados.0.smithi195.stdout:234: left oid 2 (ObjNum 100 snap 19 seq_num 100) 2024-09-16T02:14:57.962 INFO:tasks.rados.rados.0.smithi195.stdout:234: done (1 left) 2024-09-16T02:14:57.962 INFO:tasks.rados.rados.0.smithi195.stdout:237: done (0 left) 2024-09-16T02:14:57.962 INFO:tasks.rados.rados.0.smithi195.stdout:238: setattr oid 13 current snap is 19 2024-09-16T02:14:57.964 INFO:tasks.rados.rados.0.smithi195.stdout:239: snap_remove snap 13 2024-09-16T02:14:58.006 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:58.006 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:58.006 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:14:58.007 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:58.020 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:14:58.034 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:58.030+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:58.135 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:58.131+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:58.188 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:58.188 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:14:58.188 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:14:58.188 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:14:58.188 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:58.202 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:14:58.236 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:58.235+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:58.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:14:58.328 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.046083039000003, 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:14:58.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:14:58.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:14:58.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.045890264000001, 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.336 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:14:58.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:14:58.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:14:58.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.341 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.045832220999998, 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:14:58.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:14:58.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.347 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:52 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~294912,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.016964657000003, 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010472922000000001, 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.348 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 52 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308353+0000", 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308356+0000", 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3439999999999999e-06 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308602+0000", 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024608900000000001 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308617+0000", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5695000000000001e-05 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308627+0000", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.0759999999999992e-06 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308714+0000", 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7399999999999997e-05 2024-09-16T02:14:58.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308908+0000", 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019356000000000001 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.310117+0000", 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001209679 2024-09-16T02:14:58.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312215+0000", 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020981490000000001 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318705+0000", 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064893049999999999 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318826+0000", 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121625 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.016675290999999, 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.354 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:14:58.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.016622232000003, 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.361 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:14:58.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:14:58.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.981440769000002, 2024-09-16T02:14:58.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:14:58.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.7s4 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.967930144, 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0083712849999999991, 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.371 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357388+0000", 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.372 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357389+0000", 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.477e-06 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357534+0000", 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000144795 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357646+0000", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011166 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357668+0000", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1950000000000002e-05 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357901+0000", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023326999999999999 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357931+0000", 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9954000000000002e-05 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365676+0000", 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007745282 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365759+0000", 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2897000000000003e-05 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.375 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.0s6 45/32 ECSubWrite(tid=12, reqid=client.4598.0:252, at_version=0'0, trim_to=0'0, roll_forward_to=45'3))", 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.967786167, 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0081701599999999992, 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357532+0000", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357534+0000", 2024-09-16T02:14:58.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0870000000000001e-06 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357564+0000", 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0195000000000001e-05 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357660+0000", 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5581000000000005e-05 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357682+0000", 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2688999999999999e-05 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358020+0000", 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033732899999999998 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6959000000000001e-05 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365601+0000", 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0075544330000000002 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.365702+0000", 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000100887 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 45/32 ECSubWrite(tid=21, reqid=osd.10.0:21, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.955797826000001, 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010210629000000001, 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369520+0000", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369521+0000", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2500000000000005e-07 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369567+0000", 2024-09-16T02:14:58.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6436000000000001e-05 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369723+0000", 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015562700000000001 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369744+0000", 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1522000000000002e-05 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369953+0000", 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020849599999999999 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.369975+0000", 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2484999999999999e-05 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379641+0000", 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096656490000000001 2024-09-16T02:14:58.384 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379731+0000", 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9789000000000004e-05 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 45/32 ECSubWrite(tid=19, reqid=osd.8.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=45'13))", 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 23.953909813999999, 2024-09-16T02:14:58.385 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0094212150000000001, 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371408+0000", 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.386 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371410+0000", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371440+0000", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9739000000000001e-05 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371544+0000", 2024-09-16T02:14:58.387 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103787 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371566+0000", 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2673000000000001e-05 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371732+0000", 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016536799999999999 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.388 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.371756+0000", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4059000000000001e-05 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380738+0000", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0089818060000000002 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380829+0000", 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1782999999999999e-05 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.389 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.968228070999999, 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.390 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.391 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.392 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:14:58.393 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:14:58.394 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.395 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.968118833999998, 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.396 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.397 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.398 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.399 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.400 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.401 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:14:58.402 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.418854653, 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.403 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.404 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:14:58.405 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:14:58.406 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:14:58.407 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.408 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.418738926, 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.409 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.410 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:14:58.411 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.412 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.413 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.397204143, 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.414 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.415 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.416 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.943147+0000", 2024-09-16T02:14:58.417 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.382170715999999, 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0090944749999999994, 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943147+0000", 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.418 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943147+0000", 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943148+0000", 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0299999999999999e-07 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.419 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943201+0000", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3412000000000003e-05 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943378+0000", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000177133 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943402+0000", 2024-09-16T02:14:58.420 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3278999999999999e-05 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943656+0000", 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025392699999999998 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.943682+0000", 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6669000000000002e-05 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.421 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952153+0000", 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0084706279999999991 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952242+0000", 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8824000000000002e-05 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:535 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~245760,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T02:14:58.422 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:51.996969+0000", 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3283487389999999, 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0092140999999999994, 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 535 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.423 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.996969+0000", 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.996969+0000", 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.996972+0000", 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1300000000000001e-06 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.424 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.996989+0000", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6316000000000001e-05 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997024+0000", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5179999999999999e-05 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997041+0000", 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7436e-05 2024-09-16T02:14:58.425 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997179+0000", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000138107 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997192+0000", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2952999999999999e-05 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.001998+0000", 2024-09-16T02:14:58.426 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048057689999999997 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.002275+0000", 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027727999999999997 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.006183+0000", 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039079290000000001 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.427 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:536 3.6s0 3:63499ffe:::smithi19530025-18:head [read 75212~170548,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:51.997081+0000", 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3282367910000001, 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098333090000000001, 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 536 2024-09-16T02:14:58.428 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997081+0000", 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997081+0000", 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.429 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997083+0000", 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4300000000000001e-06 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997094+0000", 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1249e-05 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997115+0000", 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.126e-05 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.430 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997125+0000", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4930000000000004e-06 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997338+0000", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000213506 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997355+0000", 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6339000000000001e-05 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.431 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.003366+0000", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060117859999999999 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.003656+0000", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028980999999999999 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.006915+0000", 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003258436 2024-09-16T02:14:58.432 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:537 3.6s0 3:63499ffe:::smithi19530025-18:head [read 18069~227691,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3281703289999998, 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012782834999999999, 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:14:58.433 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 537 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:14:58.434 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997149+0000", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.203e-06 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997156+0000", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2130000000000002e-06 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997174+0000", 2024-09-16T02:14:58.435 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7422999999999999e-05 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997188+0000", 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.45e-05 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997411+0000", 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022312100000000001 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.436 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997421+0000", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9469999999999998e-06 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.006991+0000", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095694119999999994 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.007127+0000", 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013637400000000001 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:14:58.437 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.009931+0000", 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028036419999999999 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:14:58.438 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:14:58.439 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:14:58.440 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:14:58.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:14:58.441 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:14:58.435+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:14:58.441 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:14:58.441 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:14:58.472 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:14:58.472 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:14:58.502 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:14:58.503 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:14:58.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:14:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:14:58.540 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:58.534+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:58.563 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:14:58.563 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:14:58.568 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:58.568 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:58.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:14:58.596 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:14:58.597 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:14:58.629 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:14:58.629 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:14:58.641 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:58.639+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:58.655 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:14:58.655 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:14:58.688 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:14:58.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:14:58.722 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:14:58.722 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:14:58.723 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:58.724 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:58.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:14:58.742 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:14:58.739+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:14:58.754 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:14:58.755 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:14:58.782 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:14:58.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:14:58.812 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:14:58.812 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:14:58.843 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:58.839+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:14:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:14:58.842+0000 7fa87effd640 0 -- 172.21.15.3:0/838906203 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fa880224330 msgr2=0x7fa880226720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:14:58.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:14:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:14:58.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:14:58.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:14:58.904 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:58.904 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:58.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:14:58.928 INFO:tasks.rados.rados.0.smithi195.stdout:239: done (1 left) 2024-09-16T02:14:58.928 INFO:tasks.rados.rados.0.smithi195.stdout:240: snap_remove snap 18 2024-09-16T02:14:58.971 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:58.966+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:59.044 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:14:59.038+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:14:59.097 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.097 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:14:59.146 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:14:59.143+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:14:59.247 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:14:59.243+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:14:59.278 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.279 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:14:59.348 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:14:59.342+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:14:59.419 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.420 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:14:59.478 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:14:59.474+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:14:59.550 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:14:59.546+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:14:59.604 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.604 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:14:59.651 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:59.651+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:59.751 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:14:59.747+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:14:59.780 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.780 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:14:59.852 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:14:59.850+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:14:59.925 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:14:59.925 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:14:59.927 INFO:tasks.rados.rados.0.smithi195.stdout:240: done (1 left) 2024-09-16T02:14:59.927 INFO:tasks.rados.rados.0.smithi195.stdout:241: delete oid 30 current snap is 19 2024-09-16T02:14:59.927 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 13 v 121 (ObjNum 88 snap 12 seq_num 88) dirty exists 2024-09-16T02:14:59.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:14:59.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:14:59.953 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:14:59.951+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:00.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 11] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:15:00.046 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:00.046 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-16T02:15:00.046 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-16T02:15:00.046 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-16T02:15:00.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-16T02:15:00.054 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:00.051+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:00.107 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.107 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:00.155 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:00.154+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:00.170 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:00.171 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-16T02:15:00.171 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:00.250 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.250 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.257 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:00.255+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:00.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:00.358 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:00.354+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:00.386 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.387 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:00.459 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:00.459+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:00.533 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.533 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:00.560 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:00.555+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:00.652 INFO:tasks.rados.rados.0.smithi195.stdout:238: done (1 left) 2024-09-16T02:15:00.652 INFO:tasks.rados.rados.0.smithi195.stdout:241: done (0 left) 2024-09-16T02:15:00.652 INFO:tasks.rados.rados.0.smithi195.stdout:242: read oid 10 snap -1 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:242: expect (ObjNum 77 snap 8 seq_num 77) 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:243: rmattr oid 8 current snap is 19 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:243: done (1 left) 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:244: read oid 30 snap -1 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:244: expect deleted 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:245: read oid 40 snap -1 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:245: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:246: read oid 46 snap -1 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:246: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:247: copy_from oid 2 from oid 25 current snap is 19 2024-09-16T02:15:00.653 INFO:tasks.rados.rados.0.smithi195.stdout:248: setattr oid 3 current snap is 19 2024-09-16T02:15:00.658 INFO:tasks.rados.rados.0.smithi195.stdout:244: done (5 left) 2024-09-16T02:15:00.658 INFO:tasks.rados.rados.0.smithi195.stdout:249: read oid 17 snap -1 2024-09-16T02:15:00.658 INFO:tasks.rados.rados.0.smithi195.stdout:249: expect (ObjNum 97 snap 16 seq_num 97) 2024-09-16T02:15:00.658 INFO:tasks.rados.rados.0.smithi195.stdout:250: setattr oid 7 current snap is 19 2024-09-16T02:15:00.661 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:00.654+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:00.664 INFO:tasks.rados.rados.0.smithi195.stdout:242: expect (ObjNum 77 snap 8 seq_num 77) 2024-09-16T02:15:00.669 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 578402 2024-09-16T02:15:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.685 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 40744 2024-09-16T02:15:00.688 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:00.707 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.728 INFO:tasks.rados.rados.0.smithi195.stdout:245: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-16T02:15:00.728 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.736 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.748 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 635717 2024-09-16T02:15:00.748 INFO:tasks.rados.rados.0.smithi195.stdout:246: expect (ObjNum 98 snap 16 seq_num 98) 2024-09-16T02:15:00.749 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.761 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:00.759+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:00.769 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1280666 2024-09-16T02:15:00.772 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 3 v 26 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:247: finishing copy_from racing read to smithi19530025-2 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:247: finishing copy_from to smithi19530025-2 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 2 v 8 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:249: expect (ObjNum 97 snap 16 seq_num 97) 2024-09-16T02:15:00.786 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.811 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.811 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:00.836 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 2266270 2024-09-16T02:15:00.844 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:00.862 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:00.858+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:242: done (6 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:245: done (5 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:246: done (4 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:247: done (3 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:248: done (2 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:249: done (1 left) 2024-09-16T02:15:00.881 INFO:tasks.rados.rados.0.smithi195.stdout:251: setattr oid 48 current snap is 19 2024-09-16T02:15:00.887 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 4 current snap is 19 2024-09-16T02:15:00.888 INFO:tasks.rados.rados.0.smithi195.stdout:252: seq_num 105 ranges {0=606208,606208=688128,1294336=475136} 2024-09-16T02:15:00.898 INFO:tasks.rados.rados.0.smithi195.stdout:252: writing smithi19530025-4 from 0 to 606208 tid 1 2024-09-16T02:15:00.910 INFO:tasks.rados.rados.0.smithi195.stdout:252: writing smithi19530025-4 from 606208 to 1294336 tid 2 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:252: writing smithi19530025-4 from 1294336 to 1769472 tid 3 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 127 (ObjNum 93 snap 14 seq_num 93) dirty exists 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:252: finishing write tid 1 to smithi19530025-4 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:252: finishing write tid 2 to smithi19530025-4 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:251: done (2 left) 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 20 current snap is 19 2024-09-16T02:15:00.919 INFO:tasks.rados.rados.0.smithi195.stdout:253: seq_num 106 ranges {327680=622592,950272=442368,1392640=524288,1916928=196608} 2024-09-16T02:15:00.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:00.929 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:00.933 INFO:tasks.rados.rados.0.smithi195.stdout:253: writing smithi19530025-20 from 327680 to 950272 tid 1 2024-09-16T02:15:00.939 INFO:tasks.rados.rados.0.smithi195.stdout:253: writing smithi19530025-20 from 950272 to 1392640 tid 2 2024-09-16T02:15:00.945 INFO:tasks.rados.rados.0.smithi195.stdout:253: writing smithi19530025-20 from 1392640 to 1916928 tid 3 2024-09-16T02:15:00.947 INFO:tasks.rados.rados.0.smithi195.stdout:253: writing smithi19530025-20 from 1916928 to 2113536 tid 4 2024-09-16T02:15:00.947 INFO:tasks.rados.rados.0.smithi195.stdout:254: copy_from oid 40 from oid 37 current snap is 19 2024-09-16T02:15:00.947 INFO:tasks.rados.rados.0.smithi195.stdout:252: finishing write tid 3 to smithi19530025-4 2024-09-16T02:15:00.947 INFO:tasks.rados.rados.0.smithi195.stdout:255: setattr oid 27 current snap is 19 2024-09-16T02:15:00.949 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 1 to smithi19530025-20 2024-09-16T02:15:00.950 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 39 current snap is 19 2024-09-16T02:15:00.950 INFO:tasks.rados.rados.0.smithi195.stdout:256: seq_num 107 ranges {2048000=638976,2686976=458752,3145728=704512} 2024-09-16T02:15:00.959 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:00.959 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:00.963 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:00.962+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:00.973 INFO:tasks.rados.rados.0.smithi195.stdout:256: writing smithi19530025-39 from 2048000 to 2686976 tid 1 2024-09-16T02:15:00.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:00.976 INFO:tasks.rados.rados.0.smithi195.stdout:256: writing smithi19530025-39 from 2686976 to 3145728 tid 2 2024-09-16T02:15:00.982 INFO:tasks.rados.rados.0.smithi195.stdout:256: writing smithi19530025-39 from 3145728 to 3850240 tid 3 2024-09-16T02:15:00.982 INFO:tasks.rados.rados.0.smithi195.stdout:257: snap_remove snap 12 2024-09-16T02:15:01.064 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:01.058+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:01.093 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:01.094 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:01.118 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:01.119 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:01.139 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:01.139 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:01.144 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:01.144 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:01.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:01.165 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:01.158+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:01.174 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:01.174 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:01.202 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:01.202 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:01.230 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:01.231 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:01.263 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:01.264 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:01.266 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:01.263+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:01.280 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:01.280 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:01.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:01.280 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:192 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~557056] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.962918782000003, 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015517632999999999, 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 192 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.281 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314812+0000", 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029999999999999e-06 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315788+0000", 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097551599999999995 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315825+0000", 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6742000000000002e-05 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315847+0000", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2396e-05 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316018+0000", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170729 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316179+0000", 2024-09-16T02:15:01.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016178000000000001 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318923+0000", 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002743381 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071732469999999998 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330202+0000", 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004106391 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330327+0000", 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012444800000000001 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:193 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~376832] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:15:01.286 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.956316203999997, 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094258069999999996, 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:15:01.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321412+0000", 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321415+0000", 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8789999999999999e-06 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321889+0000", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047466400000000002 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321924+0000", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4134999999999998e-05 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321944+0000", 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0060000000000001e-05 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322086+0000", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014260999999999999 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322329+0000", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024320400000000001 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.324137+0000", 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018075879999999999 2024-09-16T02:15:01.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328050+0000", 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039127119999999996 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330645+0000", 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025952829999999999 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330838+0000", 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019267199999999999 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:194 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (56) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:01.292 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.955739758, 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011982552, 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 194 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321990+0000", 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.756e-06 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322010+0000", 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9443e-05 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322033+0000", 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322049+0000", 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6201999999999998e-05 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326191+0000", 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041419430000000004 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326397+0000", 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020605800000000001 2024-09-16T02:15:01.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327141+0000", 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074450000000000004 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328976+0000", 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018343109999999999 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333836+0000", 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048599389999999998 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333971+0000", 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013532799999999999 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:195 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:01.298 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.955645271999998, 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014685304, 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 195 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322084+0000", 2024-09-16T02:15:01.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322092+0000", 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6529999999999994e-06 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322111+0000", 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8909000000000001e-05 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322124+0000", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3589e-05 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327750+0000", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056258819999999996 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327883+0000", 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013255499999999999 2024-09-16T02:15:01.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334789+0000", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069061050000000001 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335013+0000", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022373400000000001 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.336768+0000", 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001755454 2024-09-16T02:15:01.303 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:242 3.8s0 3:165ce5f6:::smithi19530025-40:head [copy-from ver 0] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.942770702000001, 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011828727000000001, 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.304 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 242 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334957+0000", 2024-09-16T02:15:01.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334961+0000", 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.326e-06 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.334977+0000", 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6384000000000001e-05 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335012+0000", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4465000000000003e-05 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335033+0000", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1305999999999999e-05 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336134+0000", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011016540000000001 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336397+0000", 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026217800000000001 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.346786+0000", 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010389414 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.308 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:243 3.8s0 3:165ce5f6:::smithi19530025-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.94264325, 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01238621, 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335086+0000", 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.66e-06 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335094+0000", 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9829999999999996e-06 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335118+0000", 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3716000000000001e-05 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335135+0000", 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7102999999999999e-05 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336819+0000", 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001684051 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336943+0000", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012403199999999999 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.346988+0000", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010044599 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347205+0000", 2024-09-16T02:15:01.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021740900000000001 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347471+0000", 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026565699999999998 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(osd.15.21:1 3.8s0 3:1cf09812:::smithi19530025-15:head [assert-version v4,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.364040777, 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088887500000000008, 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913687+0000", 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913691+0000", 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5030000000000002e-06 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913715+0000", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4232000000000001e-05 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913805+0000", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0000999999999994e-05 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.913831+0000", 2024-09-16T02:15:01.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5843e-05 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.914184+0000", 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035352300000000001 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.914197+0000", 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2326e-05 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.919183+0000", 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049860640000000001 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.919371+0000", 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000188178 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.922576+0000", 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032050799999999999 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:504 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~786432] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.281699318999999, 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022327653999999999, 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 504 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996033+0000", 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1969999999999998e-06 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.997993+0000", 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019604309999999999 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998029+0000", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5525999999999998e-05 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998044+0000", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5440000000000001e-05 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998168+0000", 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012395 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998340+0000", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171619 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.000528+0000", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021878459999999998 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004445+0000", 2024-09-16T02:15:01.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003916966 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018230+0000", 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013784892 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018356+0000", 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126787 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.325 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:505 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~491520] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.274357634999999, 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022250066999999998, 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 505 2024-09-16T02:15:01.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003374+0000", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.517e-06 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004387+0000", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001013445 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004415+0000", 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-16T02:15:01.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004434+0000", 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9966000000000002e-05 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004556+0000", 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121734 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004705+0000", 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014862999999999999 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.006184+0000", 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014795570000000001 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.008703+0000", 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025187090000000001 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025506+0000", 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016802766 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025620+0000", 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114627 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.331 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:506 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~278528] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.267846896, 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016078973, 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 506 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009892+0000", 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0849999999999999e-05 2024-09-16T02:15:01.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010263+0000", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037122700000000002 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010289+0000", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5979e-05 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010303+0000", 2024-09-16T02:15:01.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3876999999999999e-05 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010423+0000", 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119824 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010635+0000", 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000212091 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.011728+0000", 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001092927 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015127+0000", 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033989139999999998 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025851+0000", 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01072415 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025960+0000", 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109134 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:507 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (57) in=7b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.267393223999999, 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016008728999999999, 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.337 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 507 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010336+0000", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.623e-06 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010345+0000", 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4500000000000004e-06 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010364+0000", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010379+0000", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.012892+0000", 2024-09-16T02:15:01.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002513286 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.013156+0000", 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026390700000000002 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014239+0000", 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00108344 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015432+0000", 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192961 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026211+0000", 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010778424999999999 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026344+0000", 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013285099999999999 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.342 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:508 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.267327484999999, 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019269792000000001, 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 508 2024-09-16T02:15:01.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010402+0000", 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010408+0000", 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7699999999999998e-06 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010425+0000", 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7603000000000001e-05 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010443+0000", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8202999999999998e-05 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014784+0000", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043404239999999998 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014911+0000", 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012705400000000001 2024-09-16T02:15:01.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027153+0000", 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012242116000000001 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027274+0000", 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012078099999999999 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.029670+0000", 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023965150000000001 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:524 3.8s0 3:1e2b38c5:::smithi19530025-2:head [read 890434~354750,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:51.691461+0000", 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.5862666680000004, 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099081989999999995, 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.348 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 524 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691461+0000", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691461+0000", 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691462+0000", 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1299999999999999e-07 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691466+0000", 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9099999999999998e-06 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691475+0000", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4299999999999995e-06 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.691483+0000", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2030000000000003e-06 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.698424+0000", 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069412320000000003 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.698552+0000", 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012808700000000001 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.701370+0000", 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028176239999999999 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.352 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:586 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3685688689999997, 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018679077999999998, 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 586 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909162+0000", 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.897e-06 2024-09-16T02:15:01.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909177+0000", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4630000000000001e-05 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909215+0000", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8588999999999999e-05 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909237+0000", 2024-09-16T02:15:01.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.211e-05 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909418+0000", 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000181003 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909440+0000", 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1271999999999999e-05 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.915451+0000", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006010858 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.917204+0000", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017530720000000001 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.920089+0000", 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002885267 2024-09-16T02:15:01.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926620+0000", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065311190000000002 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927778+0000", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001157571 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927838+0000", 2024-09-16T02:15:01.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0690000000000003e-05 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:587 3.8s0 3:165ce5f6:::smithi19530025-40:head [rollback d] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3684493069999997, 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023594436999999999, 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.359 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 587 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:01.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909280+0000", 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8129999999999998e-06 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909310+0000", 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2731999999999999e-05 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909329+0000", 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9134e-05 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909687+0000", 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035797500000000001 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909698+0000", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921510+0000", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011812049999999999 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921788+0000", 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027776500000000002 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.924415+0000", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00262737 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926686+0000", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022710270000000001 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932811+0000", 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061244309999999996 2024-09-16T02:15:01.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932873+0000", 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2630000000000002e-05 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.365 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:588 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.3683449630000002, 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026094544000000001, 2024-09-16T02:15:01.368 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 588 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909384+0000", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909391+0000", 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6159999999999998e-06 2024-09-16T02:15:01.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909409+0000", 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8335999999999999e-05 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909424+0000", 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.448e-05 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909774+0000", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035054599999999997 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909792+0000", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.77e-05 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925523+0000", 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015730929000000001 2024-09-16T02:15:01.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925651+0000", 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012825900000000001 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926311+0000", 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065990199999999997 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927500+0000", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001188599 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935426+0000", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079261060000000005 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935478+0000", 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1684000000000002e-05 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.374 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:594 3.8s0 3:1e2b38c5:::smithi19530025-2:head [append 0~524288] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.346703008, 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013176393, 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 594 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931028+0000", 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6170000000000001e-06 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931898+0000", 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086988199999999999 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931932+0000", 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3949000000000003e-05 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931948+0000", 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6353000000000001e-05 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.932115+0000", 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167207 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.932417+0000", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030156999999999997 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.935157+0000", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027404019999999999 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.943666+0000", 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085089740000000007 2024-09-16T02:15:01.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.943973+0000", 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030676400000000001 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.944201+0000", 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022867499999999999 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:596 3.8s0 3:1e2b38c5:::smithi19530025-2:head [append 0~409600] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.338065083, 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014017419999999999, 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 596 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.381 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939666+0000", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.96e-06 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940429+0000", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076330500000000004 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940458+0000", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8373000000000001e-05 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940477+0000", 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9352999999999999e-05 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.945478+0000", 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005001325 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.945625+0000", 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014625 2024-09-16T02:15:01.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.947542+0000", 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019176740000000001 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.951638+0000", 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040957040000000004 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.953483+0000", 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018450929999999999 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.953680+0000", 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000197383 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:597 3.8s0 3:1e2b38c5:::smithi19530025-2:head [setxattr _header (56) in=7b] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:01.386 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.3372229629999999, 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014074267999999999, 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 597 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:01.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940506+0000", 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1829999999999999e-06 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940514+0000", 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4399999999999999e-06 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940525+0000", 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1537e-05 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940534+0000", 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9230000000000008e-06 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.948943+0000", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084088259999999995 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.949213+0000", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026987099999999998 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.950685+0000", 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014722870000000001 2024-09-16T02:15:01.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.952884+0000", 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021985749999999999 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.954457+0000", 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001572988 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.954579+0000", 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000122638 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:598 3.8s0 3:1e2b38c5:::smithi19530025-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.3371808610000002, 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017113489999999999, 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.392 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 598 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940548+0000", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940552+0000", 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.507e-06 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940563+0000", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1177e-05 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940571+0000", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.763e-06 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955266+0000", 2024-09-16T02:15:01.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014695142 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955563+0000", 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029711000000000001 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.957661+0000", 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002097921 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.397 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.397 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:01.398 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:01.398 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:01.399 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:01.399 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:01.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:01.400 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:01.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:01.401 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:01.394+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:15:01.403 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:01.403 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:01.437 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:01.438 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:01.473 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:01.473 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:01.499 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:01.499+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:01.508 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:01.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:01.527 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:01.527 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:01.527 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:01.527 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:01.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:01.545 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:01.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:01.538+0000 7f933b59e640 0 -- 172.21.15.3:0/3699765212 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f933c225150 msgr2=0x7f933c227540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:15:01.600 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:01.598+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:01.662 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:01.663 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:01.663 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:01.663 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:01.663 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:01.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:01.701 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:01.694+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:01.785 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:01.802 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:01.798+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:01.852 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.570707314000003, 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.857 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.566484868000003, 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:15:01.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:15:01.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.566189559000001, 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006345626, 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.862 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283859+0000", 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:15:01.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283921+0000", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1947999999999997e-05 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284076+0000", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015468599999999999 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284095+0000", 2024-09-16T02:15:01.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8686e-05 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285435+0000", 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001340639 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285464+0000", 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8411999999999998e-05 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290127+0000", 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046638529999999999 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290204+0000", 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079000000000001e-05 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=1, reqid=client.4598.0:17, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.564155951000004, 2024-09-16T02:15:01.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048188320000000003, 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:01.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285894+0000", 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.667e-06 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285942+0000", 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8133999999999997e-05 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286032+0000", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9883000000000007e-05 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286080+0000", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7605000000000002e-05 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286237+0000", 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015714200000000001 2024-09-16T02:15:01.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286262+0000", 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5012e-05 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043960889999999997 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290710+0000", 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2299999999999997e-05 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.553732936000003, 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.871 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:15:01.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:15:01.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.551068555999997, 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:15:01.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.551009336, 2024-09-16T02:15:01.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:01.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:15:01.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.550602030999997, 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.885 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:15:01.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.548721286000003, 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.890 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:15:01.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.894 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.548445721999997, 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:15:01.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.547120325000002, 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00583893, 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.899 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302929+0000", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.717e-06 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302970+0000", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1564999999999999e-05 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303048+0000", 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8059000000000006e-05 2024-09-16T02:15:01.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303070+0000", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282999999999999e-05 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306196+0000", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00312663 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306213+0000", 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6747000000000001e-05 2024-09-16T02:15:01.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308730+0000", 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002516796 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308766+0000", 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6133000000000001e-05 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.499914074000003, 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046039000000000002, 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350134+0000", 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350792+0000", 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065815600000000002 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350844+0000", 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1777999999999998e-05 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350861+0000", 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6826000000000001e-05 2024-09-16T02:15:01.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351025+0000", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164623 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351055+0000", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9672999999999999e-05 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354677+0000", 2024-09-16T02:15:01.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036216830000000001 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354737+0000", 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0534e-05 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=45, reqid=client.4598.0:244, at_version=44'36, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:01.908 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.430582491999999, 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013820059000000001, 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419467+0000", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9400000000000001e-06 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419693+0000", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000226312 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431387+0000", 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011694154999999999 2024-09-16T02:15:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431404+0000", 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6945000000000001e-05 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431560+0000", 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015586199999999999 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431575+0000", 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4979e-05 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433182+0000", 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016072700000000001 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433285+0000", 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010259600000000001 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=46, reqid=client.4598.0:245, at_version=44'38, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.418965377999999, 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060040980000000002, 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:01.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431084+0000", 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431346+0000", 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026225100000000002 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434399+0000", 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030528320000000001 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434422+0000", 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3326e-05 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434546+0000", 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411200000000001 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434562+0000", 2024-09-16T02:15:01.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5075999999999999e-05 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437029+0000", 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024676699999999999 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437086+0000", 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7008e-05 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.917 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=48, reqid=client.4598.0:247, at_version=44'40, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.415026284, 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29780827999999998, 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435023+0000", 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435063+0000", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731424+0000", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29636148400000001 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731439+0000", 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.453e-05 2024-09-16T02:15:01.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731595+0000", 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015660699999999999 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731614+0000", 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8243000000000001e-05 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732758+0000", 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011446360000000001 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732829+0000", 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1191000000000003e-05 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:15:01.922 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.493153415999998, 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080490270000000003, 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356895+0000", 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356948+0000", 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3032000000000003e-05 2024-09-16T02:15:01.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357075+0000", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012766899999999999 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357100+0000", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5086e-05 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357282+0000", 2024-09-16T02:15:01.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018186299999999999 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7348999999999999e-05 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364849+0000", 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075397019999999997 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364943+0000", 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3709999999999996e-05 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.488402022999999, 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070518059999999999, 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.927 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361646+0000", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361693+0000", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6882000000000003e-05 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361805+0000", 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112046 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361834+0000", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9215999999999999e-05 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362268+0000", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043351899999999999 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362294+0000", 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6142000000000001e-05 2024-09-16T02:15:01.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368607+0000", 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063134869999999996 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9844000000000002e-05 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.486327107000001, 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059443869999999998, 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363722+0000", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.733e-06 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363805+0000", 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2776999999999997e-05 2024-09-16T02:15:01.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363906+0000", 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010156 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363931+0000", 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4966000000000001e-05 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365907+0000", 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019751790000000001 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365933+0000", 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205999999999999e-05 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369569+0000", 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036364890000000001 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369665+0000", 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5476999999999994e-05 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 62/61 ECSubWrite(tid=18, reqid=client.4598.0:512, at_version=62'109, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.834301054999999, 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048170599999999997, 2024-09-16T02:15:01.936 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015748+0000", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.934e-06 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016438+0000", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069006799999999995 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016818+0000", 2024-09-16T02:15:01.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038001400000000002 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016836+0000", 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7214000000000002e-05 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016964+0000", 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012812000000000001 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016981+0000", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7065999999999999e-05 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.020510+0000", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035291950000000002 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.020563+0000", 2024-09-16T02:15:01.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3449000000000003e-05 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 62/61 ECSubWrite(tid=19, reqid=client.4598.0:513, at_version=62'110, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.833599887, 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004743438, 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:01.941 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016449+0000", 2024-09-16T02:15:01.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6500000000000001e-06 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016746+0000", 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029719700000000002 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016886+0000", 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013917000000000001 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:01.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016902+0000", 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6633000000000002e-05 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.018291+0000", 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013883089999999999 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.018321+0000", 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0318999999999999e-05 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.021129+0000", 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002807684 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.021191+0000", 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2476000000000003e-05 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:01.945 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:01.946 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:01.947 INFO:tasks.rados.rados.0.smithi195.stdout:257: done (6 left) 2024-09-16T02:15:01.947 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 31 current snap is 19 2024-09-16T02:15:01.947 INFO:tasks.rados.rados.0.smithi195.stdout:258: seq_num 108 ranges {507904=655360,1163264=606208,1769472=475136,2244608=81920} 2024-09-16T02:15:01.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:01.949 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:01.942+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:01.955 INFO:tasks.rados.rados.0.smithi195.stdout:258: writing smithi19530025-31 from 507904 to 1163264 tid 1 2024-09-16T02:15:01.960 INFO:tasks.rados.rados.0.smithi195.stdout:258: writing smithi19530025-31 from 1163264 to 1769472 tid 2 2024-09-16T02:15:01.964 INFO:tasks.rados.rados.0.smithi195.stdout:258: writing smithi19530025-31 from 1769472 to 2244608 tid 3 2024-09-16T02:15:01.966 INFO:tasks.rados.rados.0.smithi195.stdout:258: writing smithi19530025-31 from 2244608 to 2326528 tid 4 2024-09-16T02:15:01.966 INFO:tasks.rados.rados.0.smithi195.stdout:259: snap_remove snap 17 2024-09-16T02:15:02.048 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:02.042+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:02.118 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:02.118 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:02.118 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:02.118 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:02.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:02.149 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:02.147+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:02.250 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:02.250 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:02.250 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:02.250 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:02.250 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:02.251 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:02.247+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:02.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:02.351 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:02.349+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:02.391 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 56.319080352999997, 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.392 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:15:02.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:15:02.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.814310286000001, 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020204412000000001, 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.399 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576033+0000", 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6599999999999995e-07 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576055+0000", 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1931999999999999e-05 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576066+0000", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1369000000000001e-05 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576074+0000", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5630000000000003e-06 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594265+0000", 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018191161000000001 2024-09-16T02:15:02.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594479+0000", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021409700000000001 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594685+0000", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000205708 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595917+0000", 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232564 2024-09-16T02:15:02.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596058+0000", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014070900000000001 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596135+0000", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6878000000000005e-05 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596237+0000", 2024-09-16T02:15:02.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101665 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.593003756000002, 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019987173, 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.405 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:02.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797340+0000", 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797353+0000", 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3156000000000001e-05 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797372+0000", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8382000000000001e-05 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797386+0000", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4262e-05 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811114+0000", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013728557000000001 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811354+0000", 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023977499999999999 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811632+0000", 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027806899999999998 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815429+0000", 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037971509999999999 2024-09-16T02:15:02.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817182+0000", 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001752382 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817262+0000", 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0240000000000004e-05 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817326+0000", 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4022000000000005e-05 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.592932273999999, 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020141841000000001, 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.411 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797412+0000", 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.243e-06 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797542+0000", 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077400000000001 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797562+0000", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9644e-05 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797577+0000", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812322+0000", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014744597999999999 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812456+0000", 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134466 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812671+0000", 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000214537 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815454+0000", 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027836179999999999 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817415+0000", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019603979999999999 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817490+0000", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5012000000000006e-05 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817552+0000", 2024-09-16T02:15:02.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2331999999999999e-05 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.592739569000003, 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020132852, 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:02.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797605+0000", 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7069999999999999e-06 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797620+0000", 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5585000000000001e-05 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797639+0000", 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8212000000000001e-05 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797658+0000", 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9514999999999999e-05 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813882+0000", 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016224045999999999 2024-09-16T02:15:02.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814108+0000", 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022626600000000001 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814462+0000", 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035324400000000001 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815978+0000", 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015162000000000001 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817614+0000", 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001636366 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817684+0000", 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9701000000000002e-05 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817736+0000", 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2009999999999998e-05 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:122 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.481265387999997, 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019924002999999999, 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.423 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 122 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909077+0000", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909078+0000", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.23e-07 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909084+0000", 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000002e-06 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909094+0000", 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8330000000000006e-06 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909104+0000", 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0781999999999999e-05 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922447+0000", 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013342309 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922571+0000", 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012409699999999999 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.922765+0000", 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000193869 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926685+0000", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003919952 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928862+0000", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002177216 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.928940+0000", 2024-09-16T02:15:02.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8281000000000005e-05 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.929001+0000", 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1030999999999999e-05 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:02.429 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.481224548, 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:15:02.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:15:02.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.435 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 55.481100154000003, 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.436 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:02.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:15:02.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:1 3.3s0 3:c481b30a:::smithi19530025-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.112947517000002, 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021556188, 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.442 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:02.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277399+0000", 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4400000000000001e-06 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277527+0000", 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012811100000000001 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277557+0000", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0645000000000002e-05 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277594+0000", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6848999999999999e-05 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277805+0000", 2024-09-16T02:15:02.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021101299999999999 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.278199+0000", 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039380300000000002 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.280065+0000", 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018661680000000001 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286512+0000", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064472219999999998 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298756+0000", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012244026 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298951+0000", 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194911 2024-09-16T02:15:02.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:2 3.3s0 3:c481b30a:::smithi19530025-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.112693647999997, 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022809754000000002, 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.448 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:02.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277650+0000", 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277657+0000", 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277675+0000", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8148999999999999e-05 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277691+0000", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5943e-05 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282023+0000", 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004332247 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282308+0000", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028409399999999998 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285666+0000", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033580379999999998 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.292774+0000", 2024-09-16T02:15:02.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071080379999999997 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300235+0000", 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074610759999999996 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300459+0000", 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224033 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.112528976, 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.454 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:15:02.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:15:02.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.458 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.109050582999998, 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.459 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:15:02.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:02.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.10792841, 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853072, 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.465 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282416+0000", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282453+0000", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7261000000000001e-05 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298874+0000", 2024-09-16T02:15:02.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016420846999999999 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298893+0000", 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9695e-05 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299501+0000", 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000607509 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299522+0000", 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1039e-05 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303194+0000", 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036718020000000001 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303267+0000", 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3598999999999998e-05 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.469 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.107407574, 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.470 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:15:02.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:15:02.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:02.476 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.107311355, 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:15:02.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:15:02.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.482 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:58 3.2s0 3:431ee5ca:::smithi19530025-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.075282000999998, 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021031579000000002, 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 58 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315062+0000", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315685+0000", 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062327200000000002 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315711+0000", 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5519000000000001e-05 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315731+0000", 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9775e-05 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322701+0000", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069698679999999997 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323044+0000", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034300800000000001 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324822+0000", 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017778659999999999 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.329203+0000", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043815149999999999 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.335988+0000", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067852160000000002 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.336092+0000", 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103927 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.487 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:59 3.2s0 3:431ee5ca:::smithi19530025-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.074572099999997, 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022592232, 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.488 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 59 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8729999999999999e-06 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315782+0000", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2629999999999992e-06 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315822+0000", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0095000000000002e-05 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315843+0000", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0905999999999999e-05 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326893+0000", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011050249999999999 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.327039+0000", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145723 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328011+0000", 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097276899999999996 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.330713+0000", 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027020989999999999 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338142+0000", 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074284329999999999 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338363+0000", 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022082100000000001 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 38.074472933999999, 2024-09-16T02:15:02.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:02.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.496 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:15:02.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.022737187000001, 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:02.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.018449507, 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:15:02.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:02.506 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:02.508 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:02.508 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:02.507+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:02.609 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:02.607+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:02.639 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:02.639 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:02.639 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:02.639 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:02.653 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:02.710 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:02.707+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:02.772 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:02.772 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:02.773 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:02.773 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:02.773 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:02.786 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:02.810 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:02.807+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:02.912 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 54.719226507000002, 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:15:02.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.916 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:15:02.917 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.734200270000002, 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066455200000000002, 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175561+0000", 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7030000000000002e-06 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175575+0000", 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3889e-05 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175608+0000", 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3704000000000001e-05 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175627+0000", 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175763+0000", 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013625000000000001 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175774+0000", 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.116e-05 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179478+0000", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037039989999999999 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179657+0000", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179538 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179882+0000", 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022455700000000001 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181213+0000", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013309039999999999 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182065+0000", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085171900000000002 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182143+0000", 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7904999999999994e-05 2024-09-16T02:15:02.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182203+0000", 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0909000000000002e-05 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 38/32 ECSubWrite(tid=14, reqid=client.4598.0:197, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 35.582617908000003, 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058517350000000003, 2024-09-16T02:15:02.924 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327142+0000", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.59e-06 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327187+0000", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5467999999999999e-05 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327289+0000", 2024-09-16T02:15:02.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102086 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327312+0000", 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2849000000000001e-05 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327454+0000", 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014216700000000001 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.927 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327467+0000", 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009e-05 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332903+0000", 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054355560000000002 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332992+0000", 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9010000000000003e-05 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.928 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.566635075000001, 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095323660000000004, 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343124+0000", 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.06e-06 2024-09-16T02:15:02.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343143+0000", 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8842e-05 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343322+0000", 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179414 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343345+0000", 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2629e-05 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343569+0000", 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022398900000000001 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343596+0000", 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6489e-05 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352536+0000", 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089404250000000001 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352656+0000", 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119518 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.933 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/32 ECSubWrite(tid=19, reqid=osd.6.0:19, at_version=44'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.566545827999999, 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083893449999999994, 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343214+0000", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343264+0000", 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0175e-05 2024-09-16T02:15:02.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343378+0000", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114159 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343396+0000", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7926000000000001e-05 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343513+0000", 2024-09-16T02:15:02.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117056 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343528+0000", 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4324999999999999e-05 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351501+0000", 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079735710000000005 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.937 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351602+0000", 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010059 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.565210917000002, 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082631800000000002, 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.938 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344549+0000", 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344608+0000", 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8953999999999998e-05 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344655+0000", 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7268999999999998e-05 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344677+0000", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1382e-05 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344840+0000", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016295099999999999 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344865+0000", 2024-09-16T02:15:02.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5035e-05 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352714+0000", 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078496989999999999 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352810+0000", 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6146999999999999e-05 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.942 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.562395571, 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009462359, 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:02.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347364+0000", 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.623e-06 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347386+0000", 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1667000000000001e-05 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347450+0000", 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4116999999999996e-05 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347464+0000", 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4347e-05 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348099+0000", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063462999999999996 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348122+0000", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2866000000000001e-05 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356742+0000", 2024-09-16T02:15:02.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086204220000000008 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356825+0000", 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2687000000000003e-05 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.9s6 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.560672982, 2024-09-16T02:15:02.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0076692430000000001, 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:02.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349127+0000", 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1560999999999997e-05 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349203+0000", 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6447999999999997e-05 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349306+0000", 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102413 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349329+0000", 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349580+0000", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000250874 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349604+0000", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4253e-05 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356660+0000", 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070559610000000004 2024-09-16T02:15:02.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356754+0000", 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4114e-05 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 45/32 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=45'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.558485693000002, 2024-09-16T02:15:02.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010034855000000001, 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:02.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351322+0000", 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351430+0000", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108572 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351458+0000", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8229e-05 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351962+0000", 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050377899999999999 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351989+0000", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6322e-05 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361119+0000", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091308310000000007 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361307+0000", 2024-09-16T02:15:02.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188054 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 51/32 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=51'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.496154808, 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075541130000000003, 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413604+0000", 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:02.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413653+0000", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8726000000000001e-05 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414322+0000", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066915500000000005 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414345+0000", 2024-09-16T02:15:02.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2863000000000001e-05 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414490+0000", 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014519199999999999 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414515+0000", 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5312000000000001e-05 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421073+0000", 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065573719999999997 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421158+0000", 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4690000000000004e-05 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:394 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:02.961 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.002437733000001, 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094620629999999997, 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 394 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907323+0000", 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5399999999999998e-06 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907336+0000", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2605999999999999e-05 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907364+0000", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8538999999999999e-05 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907387+0000", 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979000000000001e-05 2024-09-16T02:15:02.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907499+0000", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011208 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907666+0000", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016685800000000001 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.909169+0000", 2024-09-16T02:15:02.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015026659999999999 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914888+0000", 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057194580000000002 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916546+0000", 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001657911 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916783+0000", 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236426 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:395 3.fs0 3:f97ec54c:::smithi19530025-26:head [rollback 8] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.002333423, 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011190179, 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.967 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 395 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:02.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907426+0000", 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907433+0000", 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907453+0000", 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907485+0000", 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2066000000000001e-05 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910164+0000", 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026789510000000002 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.970 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910300+0000", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013539300000000001 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.911564+0000", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001264577 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.917774+0000", 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062099729999999997 2024-09-16T02:15:02.971 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918506+0000", 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073229000000000002 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918615+0000", 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108527 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.972 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:396 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.002247368999999, 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011749021, 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 396 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.973 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907512+0000", 2024-09-16T02:15:02.974 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1060000000000001e-06 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907519+0000", 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.917e-06 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907537+0000", 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.975 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907552+0000", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912502+0000", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049500600000000001 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912741+0000", 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023893499999999999 2024-09-16T02:15:02.976 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914057+0000", 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013153119999999999 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918884+0000", 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048277240000000003 2024-09-16T02:15:02.977 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919157+0000", 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306100000000001 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919260+0000", 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102594 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.978 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 67/32 ECSubWrite(tid=79, reqid=client.4598.0:586, at_version=67'51, trim_to=0'0, roll_forward_to=66'49))", 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.920324+0000", 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.9894338769999997, 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048823690000000001, 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920324+0000", 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.979 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920324+0000", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920325+0000", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1600000000000001e-07 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920369+0000", 2024-09-16T02:15:02.980 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4013e-05 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920472+0000", 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102848 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920490+0000", 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8119999999999999e-05 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.981 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920572+0000", 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1872999999999994e-05 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.920583+0000", 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1275e-05 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.925181+0000", 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0045978570000000003 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.982 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.925207+0000", 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5867000000000002e-05 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.6s5 73/32 ECSubWrite(tid=40, reqid=osd.15.0:40, at_version=73'23, trim_to=0'0, roll_forward_to=60'21))", 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.945360+0000", 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.96439864500000005, 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.005288091, 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.983 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945360+0000", 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945360+0000", 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.984 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945360+0000", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9699999999999995e-07 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945408+0000", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7902999999999999e-05 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945489+0000", 2024-09-16T02:15:02.985 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1213999999999999e-05 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945513+0000", 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3068999999999999e-05 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945749+0000", 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236011 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.986 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945777+0000", 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8467999999999999e-05 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.950567+0000", 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047894920000000002 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.950648+0000", 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1137000000000006e-05 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.987 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 73/32 ECSubWrite(tid=86, reqid=osd.6.0:86, at_version=73'70, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.945735+0000", 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.96402362399999997, 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049030080000000004, 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.988 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945735+0000", 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945735+0000", 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945736+0000", 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.077e-06 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.989 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945786+0000", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0173000000000002e-05 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945900+0000", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011377199999999999 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.945923+0000", 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3626e-05 2024-09-16T02:15:02.990 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.946104+0000", 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000181187 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.946129+0000", 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4819000000000001e-05 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.991 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.950534+0000", 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044046069999999996 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.950638+0000", 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010374699999999999 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 73/32 ECSubWrite(tid=106, reqid=osd.5.0:105, at_version=0'0, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.954370+0000", 2024-09-16T02:15:02.992 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.95538826200000004, 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050954909999999997, 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954370+0000", 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.993 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954370+0000", 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954372+0000", 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.689e-06 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954400+0000", 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8299000000000001e-05 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.994 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954480+0000", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0338000000000002e-05 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954502+0000", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1645e-05 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954738+0000", 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023634400000000001 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.995 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954764+0000", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5612000000000001e-05 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.959372+0000", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0046078109999999999 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.959465+0000", 2024-09-16T02:15:02.996 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3752999999999995e-05 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 73/32 ECSubWrite(tid=107, reqid=osd.5.0:107, at_version=73'75, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.956526+0000", 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.95323257800000005, 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0056851540000000004, 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:02.997 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956526+0000", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956526+0000", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956528+0000", 2024-09-16T02:15:02.998 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.86e-06 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956700+0000", 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017251500000000001 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956812+0000", 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000111595 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:02.999 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.956842+0000", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0207999999999999e-05 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.957007+0000", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000165143 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.957033+0000", 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6234e-05 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.000 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962119+0000", 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0050856419999999996 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962211+0000", 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1957000000000001e-05 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.001 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.94845934499999995, 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067590630000000001, 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.002 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961300+0000", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8999999999999995e-07 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961666+0000", 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036643500000000002 2024-09-16T02:15:03.003 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961786+0000", 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120192 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961811+0000", 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4405000000000001e-05 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.004 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962023+0000", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000212489 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962050+0000", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6968e-05 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.967810+0000", 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057602699999999996 2024-09-16T02:15:03.005 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968058+0000", 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000247514 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/66 ECSubWrite(tid=104, reqid=client.4598.0:647, at_version=73'29, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.967990+0000", 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 0.941768618, 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053139270000000004, 2024-09-16T02:15:03.006 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.967990+0000", 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.967990+0000", 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.007 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.967992+0000", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9700000000000002e-06 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968287+0000", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00029584700000000002 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968384+0000", 2024-09-16T02:15:03.008 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6413e-05 2024-09-16T02:15:03.009 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.009 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.009 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.009 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968407+0000", 2024-09-16T02:15:03.009 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3575e-05 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968586+0000", 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000178092 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.012 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968613+0000", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7940999999999999e-05 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.973205+0000", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0045919389999999997 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.973304+0000", 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.815e-05 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.013 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.014 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.014 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.014 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.014 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:02.906+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:03.015 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:03.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:03.017 INFO:tasks.rados.rados.0.smithi195.stdout:259: done (7 left) 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:260: rmattr oid 33 current snap is 19 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 2 to smithi19530025-20 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 3 to smithi19530025-20 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 4 to smithi19530025-20 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 27 v 21 (ObjNum 33 snap 0 seq_num 3150912496) dirty exists 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: finishing write tid 4 to smithi19530025-4 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: finishing write tid 5 to smithi19530025-4 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: oid 4 updating version 0 to 50 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: oid 4 updating version 50 to 51 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: oid 4 updating version 51 to 52 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: oid 4 updating version 52 to 53 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 4 v 53 (ObjNum 105 snap 19 seq_num 105) dirty exists 2024-09-16T02:15:03.018 INFO:tasks.rados.rados.0.smithi195.stdout:252: left oid 4 (ObjNum 105 snap 19 seq_num 105) 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:256: finishing write tid 1 to smithi19530025-39 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:256: finishing write tid 2 to smithi19530025-39 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:256: finishing write tid 3 to smithi19530025-39 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:254: finishing copy_from racing read to smithi19530025-40 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:254: finishing copy_from to smithi19530025-40 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 40 v 64 (ObjNum 99 snap 16 seq_num 99) dirty exists 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 5 to smithi19530025-20 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: finishing write tid 6 to smithi19530025-20 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: oid 20 updating version 0 to 77 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: oid 20 version 77 is already newer than 75 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: oid 20 version 77 is already newer than 74 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: oid 20 updating version 77 to 78 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:253: oid 20 version 78 is already newer than 76 2024-09-16T02:15:03.019 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 20 v 78 (ObjNum 106 snap 19 seq_num 106) dirty exists 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:253: left oid 20 (ObjNum 106 snap 19 seq_num 106) 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: finishing write tid 4 to smithi19530025-39 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: finishing write tid 5 to smithi19530025-39 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: oid 39 updating version 0 to 37 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: oid 39 version 37 is already newer than 35 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: oid 39 version 37 is already newer than 36 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: oid 39 updating version 37 to 38 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 39 v 38 (ObjNum 107 snap 19 seq_num 107) dirty exists 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:256: left oid 39 (ObjNum 107 snap 19 seq_num 107) 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 25 (ObjNum 89 snap 12 seq_num 89) dirty exists 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 1 to smithi19530025-31 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 2 to smithi19530025-31 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 3 to smithi19530025-31 2024-09-16T02:15:03.020 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 4 to smithi19530025-31 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 5 to smithi19530025-31 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: finishing write tid 6 to smithi19530025-31 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: oid 31 updating version 0 to 29 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: oid 31 updating version 29 to 31 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: oid 31 updating version 31 to 32 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: oid 31 version 32 is already newer than 28 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: oid 31 version 32 is already newer than 30 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 32 (ObjNum 108 snap 19 seq_num 108) dirty exists 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:258: left oid 31 (ObjNum 108 snap 19 seq_num 108) 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:250: done (7 left) 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:252: done (6 left) 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:253: done (5 left) 2024-09-16T02:15:03.021 INFO:tasks.rados.rados.0.smithi195.stdout:254: done (4 left) 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:255: done (3 left) 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:256: done (2 left) 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:258: done (1 left) 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:261: rollback oid 45 current snap is 19 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 45 to 11 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:262: snap_create 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:261: finishing rollback tid 0 to smithi19530025-45 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:261: finishing rollback tid 1 to smithi19530025-45 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 33 v 44 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:261: finishing rollback tid 2 to smithi19530025-45 2024-09-16T02:15:03.022 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 45 v 83 (ObjNum 75 snap 8 seq_num 75) dirty exists 2024-09-16T02:15:03.023 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:03.010+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:03.117 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:03.115+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:03.189 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:03.189 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:03.189 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:03.189 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.203 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:03.218 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:03.214+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:03.319 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:03.314+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:03.324 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:03.324 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:03.324 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:03.324 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:03.324 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:03.421 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:03.417+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:03.460 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:03.460 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:196 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.135849280000002, 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013336756999999999, 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.461 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 196 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:03.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321917+0000", 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6970000000000002e-06 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321944+0000", 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7226e-05 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321985+0000", 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1551000000000002e-05 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322009+0000", 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3269e-05 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322176+0000", 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016762 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322357+0000", 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000180805 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.324174+0000", 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001817113 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330068+0000", 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058941790000000003 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335031+0000", 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049623899999999997 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335251+0000", 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021990700000000001 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.466 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:197 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.135706579000001, 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013892593, 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.467 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322058+0000", 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.643e-06 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322069+0000", 2024-09-16T02:15:03.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.044e-05 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322090+0000", 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1455e-05 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322106+0000", 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6163000000000001e-05 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325364+0000", 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032580809999999999 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325616+0000", 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025181900000000002 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327103+0000", 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014866549999999999 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330072+0000", 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029688729999999999 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335817+0000", 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057450959999999999 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.472 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335949+0000", 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132368 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:198 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.135612186000003, 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014215244, 2024-09-16T02:15:03.473 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 198 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322152+0000", 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3829999999999999e-06 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322159+0000", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5660000000000003e-06 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322179+0000", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9613e-05 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322190+0000", 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1542000000000001e-05 2024-09-16T02:15:03.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328227+0000", 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060372159999999998 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328485+0000", 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025810200000000002 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.329430+0000", 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094416499999999996 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330786+0000", 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013565509999999999 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336208+0000", 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054220529999999996 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336366+0000", 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015805300000000001 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 44/32 ECSubWrite(tid=187, reqid=osd.13.0:187, at_version=44'26, trim_to=0'0, roll_forward_to=43'24))", 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.346793+0000", 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.110970591000001, 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075683629999999998, 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.479 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.346793+0000", 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.346793+0000", 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.346793+0000", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8700000000000005e-07 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.346876+0000", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2667e-05 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.346991+0000", 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011455600000000001 2024-09-16T02:15:03.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.347019+0000", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8674999999999999e-05 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.347181+0000", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016190399999999999 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.347205+0000", 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3583000000000001e-05 2024-09-16T02:15:03.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354281+0000", 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070765569999999998 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.354361+0000", 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9734000000000006e-05 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/32 ECSubWrite(tid=42, reqid=osd.11.0:42, at_version=44'33, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.349770+0000", 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.107993539999999, 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0081907360000000005, 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349770+0000", 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349770+0000", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349770+0000", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8999999999999996e-07 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349822+0000", 2024-09-16T02:15:03.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1824999999999999e-05 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349970+0000", 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000147787 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.349994+0000", 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4436000000000001e-05 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350167+0000", 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017213999999999999 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350192+0000", 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5746e-05 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.357852+0000", 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0076598889999999996 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.357960+0000", 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108223 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/32 ECSubWrite(tid=43, reqid=osd.11.0:43, at_version=44'35, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 30.107215931999999, 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010145988, 2024-09-16T02:15:03.488 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:03.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350559+0000", 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1813e-05 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350610+0000", 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1325000000000001e-05 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350717+0000", 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106846 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350739+0000", 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2181999999999999e-05 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352445+0000", 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017059110000000001 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352470+0000", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4185999999999998e-05 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.360599+0000", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0081295510000000005 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.360693+0000", 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4173999999999997e-05 2024-09-16T02:15:03.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.fs5 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.350749+0000", 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.107014392, 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062926350000000004, 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350749+0000", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350749+0000", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350750+0000", 2024-09-16T02:15:03.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.406e-06 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350805+0000", 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4781999999999998e-05 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350846+0000", 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0472000000000001e-05 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350867+0000", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1458999999999999e-05 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.350999+0000", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013214499999999999 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351021+0000", 2024-09-16T02:15:03.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2359999999999999e-05 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356947+0000", 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0059254609999999999 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.357041+0000", 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4549999999999994e-05 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.497 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.097614431, 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082887169999999993, 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360150+0000", 2024-09-16T02:15:03.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9000000000000005e-07 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360200+0000", 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9781000000000003e-05 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360340+0000", 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014055900000000001 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360360+0000", 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0205999999999999e-05 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360530+0000", 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000169414 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360554+0000", 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3828999999999998e-05 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.368346+0000", 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.007792902 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.368438+0000", 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1036e-05 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.09594135, 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095998799999999999, 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361897+0000", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4048000000000007e-05 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361976+0000", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9432999999999997e-05 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361998+0000", 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1996000000000001e-05 2024-09-16T02:15:03.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362235+0000", 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023705199999999999 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362259+0000", 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3782e-05 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371349+0000", 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090897349999999998 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371422+0000", 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3257999999999996e-05 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:03.507 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.095082957999999, 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086860159999999995, 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362682+0000", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0839999999999999e-06 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362739+0000", 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6721000000000001e-05 2024-09-16T02:15:03.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362788+0000", 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8875000000000002e-05 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362811+0000", 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3275e-05 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362968+0000", 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015707199999999999 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362993+0000", 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4865999999999999e-05 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371258+0000", 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082643939999999996 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371366+0000", 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108729 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.092905698999999, 2024-09-16T02:15:03.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091943479999999998, 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:03.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364860+0000", 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1169999999999998e-06 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365009+0000", 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000149808 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365058+0000", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8677999999999997e-05 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365081+0000", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2486000000000001e-05 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365254+0000", 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017369400000000001 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365280+0000", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5505000000000001e-05 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.373957+0000", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00867712 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.374052+0000", 2024-09-16T02:15:03.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4939999999999996e-05 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 51/32 ECSubWrite(tid=224, reqid=osd.13.0:224, at_version=51'45, trim_to=0'0, roll_forward_to=49'43))", 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.414000+0000", 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 23.043763721000001, 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0064978830000000003, 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414000+0000", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414000+0000", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414000+0000", 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9999999999999996e-07 2024-09-16T02:15:03.518 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414057+0000", 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6907999999999998e-05 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.417103+0000", 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030459770000000001 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.519 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.417126+0000", 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2444999999999999e-05 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.417350+0000", 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022403700000000001 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.417376+0000", 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6029000000000001e-05 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.520 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.420415+0000", 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030395940000000001 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.420497+0000", 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2093000000000002e-05 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.521 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 52/32 ECSubWrite(tid=226, reqid=client.4598.0:412, at_version=52'47, trim_to=0'0, roll_forward_to=51'45))", 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.994649+0000", 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.463114284, 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0068458670000000003, 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.994649+0000", 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.522 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.994649+0000", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.994650+0000", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.44e-06 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.995248+0000", 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00059766500000000005 2024-09-16T02:15:03.523 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.995325+0000", 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6810000000000005e-05 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.995345+0000", 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0256000000000001e-05 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.524 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.995511+0000", 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016627499999999999 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.995541+0000", 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9556e-05 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:41.001414+0000", 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058728239999999996 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.525 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:41.001495+0000", 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1040999999999999e-05 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 57/32 ECSubWrite(tid=117, reqid=osd.11.0:117, at_version=57'104, trim_to=0'0, roll_forward_to=56'100))", 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.525487558999998, 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013673405, 2024-09-16T02:15:03.526 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.527 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932278+0000", 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9439999999999999e-06 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932320+0000", 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2052000000000003e-05 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.528 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932390+0000", 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0724999999999998e-05 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932406+0000", 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.529 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932582+0000", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000176632 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932609+0000", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6389000000000001e-05 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.943840+0000", 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011230795 2024-09-16T02:15:03.530 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.945949+0000", 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021094719999999998 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:494 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.500700222999999, 2024-09-16T02:15:03.531 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020145202000000001, 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 494 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.532 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957066+0000", 2024-09-16T02:15:03.533 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0340000000000001e-06 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957080+0000", 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3696000000000001e-05 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957110+0000", 2024-09-16T02:15:03.534 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0252000000000001e-05 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957135+0000", 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4586000000000001e-05 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957642+0000", 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050762300000000002 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.535 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957870+0000", 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022741699999999999 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.960012+0000", 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002142793 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.536 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966293+0000", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062808969999999997 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977023+0000", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010729446 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977208+0000", 2024-09-16T02:15:03.537 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000185458 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:495 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback d] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.500580446000001, 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021622549000000001, 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.538 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 495 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.539 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957185+0000", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957191+0000", 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:03.540 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957211+0000", 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0069e-05 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957229+0000", 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7856e-05 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.541 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961650+0000", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044206920000000004 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961911+0000", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026131599999999999 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.963434+0000", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00152287 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.542 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966303+0000", 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028691599999999999 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978597+0000", 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012293652 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978805+0000", 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000208304 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.543 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:496 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 16.500477922000002, 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021768414, 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 496 2024-09-16T02:15:03.544 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.545 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957287+0000", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957294+0000", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7800000000000003e-06 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957313+0000", 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9669000000000001e-05 2024-09-16T02:15:03.546 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957328+0000", 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4847e-05 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964542+0000", 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0072143199999999998 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.547 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964700+0000", 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157343 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.965510+0000", 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080991199999999998 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.968798+0000", 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032880489999999999 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.548 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978963+0000", 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010165014 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.979054+0000", 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1037000000000003e-05 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.549 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:583 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 0~3407872,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.5484917439999997, 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011937642, 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 583 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.550 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909274+0000", 2024-09-16T02:15:03.551 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8499999999999998e-06 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2686e-05 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909317+0000", 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9986999999999999e-05 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.552 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909339+0000", 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2305000000000001e-05 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909427+0000", 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7862999999999995e-05 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909520+0000", 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2571000000000005e-05 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.553 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.921209+0000", 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011689379999999999 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:584 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 1157777~2250095,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.5483728670000003, 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012758666, 2024-09-16T02:15:03.554 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 584 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.555 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909392+0000", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9369999999999998e-06 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909401+0000", 2024-09-16T02:15:03.556 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4300000000000006e-06 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909422+0000", 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1022999999999998e-05 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909438+0000", 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5739e-05 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.557 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909938+0000", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050038899999999996 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.910047+0000", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010885199999999999 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.922149+0000", 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012102296 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.558 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:585 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 1560470~1847402,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 7.5482986649999999, 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013504089, 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:03.559 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 585 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:03.560 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909466+0000", 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3829999999999999e-06 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909477+0000", 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0567000000000001e-05 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:03.561 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909494+0000", 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7796e-05 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909508+0000", 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3607e-05 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.912332+0000", 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028238 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.562 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.912425+0000", 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3457999999999999e-05 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.922969+0000", 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010543478 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:03.563 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.565 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:03.566 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:03.558+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:03.666 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:03.665+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:03.692 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:03.692 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:03.692 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:03.692 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.706 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:15:03.766 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:03.762+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:03.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:03.829 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:03.830 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:03.830 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:03.843 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:15:03.867 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:03.862+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:03.955 INFO:tasks.rados.rados.0.smithi195.stdout:260: done (2 left) 2024-09-16T02:15:03.955 INFO:tasks.rados.rados.0.smithi195.stdout:261: done (1 left) 2024-09-16T02:15:03.955 INFO:tasks.rados.rados.0.smithi195.stdout:262: done (0 left) 2024-09-16T02:15:03.955 INFO:tasks.rados.rados.0.smithi195.stdout:263: rollback oid 30 current snap is 20 2024-09-16T02:15:03.955 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 30 to 16 2024-09-16T02:15:03.956 INFO:tasks.rados.rados.0.smithi195.stdout:264: delete oid 31 current snap is 20 2024-09-16T02:15:03.967 INFO:tasks.rados.rados.0.smithi195.stdout:264: done (1 left) 2024-09-16T02:15:03.967 INFO:tasks.rados.rados.0.smithi195.stdout:265: snap_create 2024-09-16T02:15:03.967 INFO:tasks.rados.rados.0.smithi195.stdout:263: finishing rollback tid 1 to smithi19530025-30 2024-09-16T02:15:03.968 INFO:tasks.rados.rados.0.smithi195.stdout:263: finishing rollback tid 2 to smithi19530025-30 2024-09-16T02:15:03.968 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 92 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-16T02:15:03.968 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:03.967+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:03.988 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:03.988 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:04.013 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:04.013 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:10 3.5s0 3:aae22120:::smithi19530025-4:head [append 0~65536,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.278841+0000", 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.732156861, 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013808462000000001, 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.014 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 10 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.278841+0000", 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.278841+0000", 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.015 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.278849+0000", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3529999999999997e-06 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279253+0000", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040422800000000002 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279282+0000", 2024-09-16T02:15:04.016 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9369e-05 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279300+0000", 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7689e-05 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279477+0000", 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017698200000000001 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.017 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279880+0000", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040302699999999998 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.282132+0000", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002252605 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.292239+0000", 2024-09-16T02:15:04.018 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010106894999999999 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.292424+0000", 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018487399999999999 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.292650+0000", 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022544000000000001 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.019 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:11 3.5s0 3:aae22120:::smithi19530025-4:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.279343+0000", 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.731655490999998, 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.022329023999999999, 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.020 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 11 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279343+0000", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.021 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279343+0000", 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279344+0000", 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.59e-06 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279354+0000", 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.713e-06 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.022 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279394+0000", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0108000000000003e-05 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279408+0000", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3606e-05 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.285769+0000", 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0063614329999999997 2024-09-16T02:15:04.023 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.286082+0000", 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031326200000000002 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.288589+0000", 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002506936 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.024 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301241+0000", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012651661999999999 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301422+0000", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018115500000000001 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301672+0000", 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000249559 2024-09-16T02:15:04.025 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:12 3.5s0 3:aae22120:::smithi19530025-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.279433+0000", 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.731565085, 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.048412415, 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.026 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 12 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279433+0000", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279433+0000", 2024-09-16T02:15:04.027 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279434+0000", 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1999999999999999e-06 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279441+0000", 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8589999999999999e-06 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.028 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279461+0000", 2024-09-16T02:15:04.029 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9839000000000001e-05 2024-09-16T02:15:04.033 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279475+0000", 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3906999999999999e-05 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.291979+0000", 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012504479000000001 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.034 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.292204+0000", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000225186 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301779+0000", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095741890000000003 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302009+0000", 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000230406 2024-09-16T02:15:04.035 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327845+0000", 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.025836350000000001 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:13 3.5s0 3:a92536ab:::smithi19530025-5:head [append 0~81920,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.279490+0000", 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.731508235, 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.039403769999999998, 2024-09-16T02:15:04.036 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 13 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.037 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279490+0000", 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279490+0000", 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0699999999999996e-07 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.038 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.279920+0000", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042980999999999998 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280131+0000", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021091999999999999 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280149+0000", 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8068999999999999e-05 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.039 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.292949+0000", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01280005 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293307+0000", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035702900000000003 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.295396+0000", 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020897229999999999 2024-09-16T02:15:04.040 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316160+0000", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020763588999999999 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318513+0000", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002353501 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318894+0000", 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00038027200000000002 2024-09-16T02:15:04.041 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:14 3.5s0 3:a92536ab:::smithi19530025-5:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.280179+0000", 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.730818833000001, 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.044998446999999997, 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.042 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 14 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280179+0000", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280179+0000", 2024-09-16T02:15:04.043 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280181+0000", 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6929999999999999e-06 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280206+0000", 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5041999999999998e-05 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.044 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280229+0000", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2782e-05 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280244+0000", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5443000000000001e-05 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.296877+0000", 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016633193000000001 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.045 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297099+0000", 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000221487 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299578+0000", 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024795799999999999 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.046 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316163+0000", 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016584452 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324967+0000", 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088044979999999991 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.325178+0000", 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021027700000000001 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.047 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:15 3.5s0 3:a92536ab:::smithi19530025-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.280268+0000", 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.730730024000003, 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.054160166000000003, 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.048 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 15 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280268+0000", 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280268+0000", 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.049 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280270+0000", 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.61e-06 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280278+0000", 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8660000000000006e-06 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.050 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280297+0000", 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9799000000000001e-05 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280311+0000", 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3506000000000001e-05 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.300212+0000", 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019901499999999999 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.051 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.300421+0000", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020882999999999999 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327947+0000", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.027526115 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.328108+0000", 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000161046 2024-09-16T02:15:04.052 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334428+0000", 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0063198940000000004 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:22 3.5s0 3:a7785c58:::smithi19530025-8:head [append 0~131072,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.280334+0000", 2024-09-16T02:15:04.053 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.730664476999998, 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.065692312000000003, 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 22 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.054 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280334+0000", 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280334+0000", 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280335+0000", 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5260000000000001e-06 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.055 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280540+0000", 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020440399999999999 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280556+0000", 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6409000000000002e-05 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.056 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280572+0000", 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5875999999999998e-05 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.331339+0000", 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.050766985000000001 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.331638+0000", 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00029878899999999999 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.057 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332961+0000", 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001323315 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336155+0000", 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0031938190000000001 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.058 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.345862+0000", 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0097071060000000001 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.346026+0000", 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016408300000000001 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.059 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:23 3.5s0 3:a7785c58:::smithi19530025-8:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.280588+0000", 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.730409725999998, 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.066447170999999999, 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 23 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.060 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280588+0000", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280588+0000", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280589+0000", 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9000000000000005e-07 2024-09-16T02:15:04.061 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280594+0000", 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4560000000000002e-06 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280603+0000", 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6430000000000007e-06 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.062 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280611+0000", 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.153e-06 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334509+0000", 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.053898052000000002 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334673+0000", 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016388500000000001 2024-09-16T02:15:04.063 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.335627+0000", 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00095437300000000001 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.346845+0000", 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01121838 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.064 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.346924+0000", 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8582000000000001e-05 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.347036+0000", 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011165700000000001 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:24 3.5s0 3:a7785c58:::smithi19530025-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:04.065 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.280621+0000", 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.730377218000001, 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.067797500999999996, 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 24 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280621+0000", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280621+0000", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280622+0000", 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.54e-07 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280625+0000", 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.72e-06 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280635+0000", 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2289999999999992e-06 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.280642+0000", 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2230000000000001e-06 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336398+0000", 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055756093999999999 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336571+0000", 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017349600000000001 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.347082+0000", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010510548999999999 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.347174+0000", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1774000000000006e-05 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.348418+0000", 2024-09-16T02:15:04.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012445620000000001 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=2, last_complete=35'3, committed=1, applied=1))", 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.299125+0000", 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.711873048000001, 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017447341000000002, 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.071 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299125+0000", 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299125+0000", 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299127+0000", 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0760000000000001e-06 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299140+0000", 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2823000000000001e-05 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299221+0000", 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1433000000000005e-05 2024-09-16T02:15:04.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299242+0000", 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0171999999999999e-05 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316519+0000", 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01727743 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316572+0000", 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3406999999999999e-05 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=2, last_complete=35'3, committed=1, applied=1))", 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.299550+0000", 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.711448441999998, 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016616694000000001, 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.075 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299550+0000", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299550+0000", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299551+0000", 2024-09-16T02:15:04.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.497e-06 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299561+0000", 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6360000000000006e-06 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299625+0000", 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4467000000000005e-05 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299644+0000", 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.842e-05 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316104+0000", 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016460571 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.078 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316166+0000", 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.2102999999999998e-05 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302065+0000", 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.708933489000003, 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016179934, 2024-09-16T02:15:04.079 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302065+0000", 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302065+0000", 2024-09-16T02:15:04.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302066+0000", 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.823e-06 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302078+0000", 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1372999999999999e-05 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302153+0000", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5712999999999998e-05 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302173+0000", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9656e-05 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318202+0000", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016028938 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.318245+0000", 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2431000000000001e-05 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=2, last_complete=35'3, committed=1, applied=1))", 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302525+0000", 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.708473071, 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014215966, 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302525+0000", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302525+0000", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302526+0000", 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.094e-06 2024-09-16T02:15:04.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302540+0000", 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4032000000000001e-05 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.303858+0000", 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013175789999999999 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.303874+0000", 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6665999999999999e-05 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316684+0000", 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012809991999999999 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316741+0000", 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6603e-05 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.086 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302583+0000", 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.708415254000002, 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015099679, 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302583+0000", 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302583+0000", 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302585+0000", 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.097e-06 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302597+0000", 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2072999999999999e-05 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302677+0000", 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9676000000000001e-05 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302695+0000", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8808999999999999e-05 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317625+0000", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014929649999999999 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317682+0000", 2024-09-16T02:15:04.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7374000000000003e-05 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.304289+0000", 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.706709418999999, 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020324175999999999, 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304289+0000", 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304289+0000", 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304291+0000", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1629999999999999e-06 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304303+0000", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2189e-05 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304400+0000", 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7442999999999995e-05 2024-09-16T02:15:04.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304421+0000", 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0948999999999999e-05 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324571+0000", 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020149518000000002 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324613+0000", 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1913999999999999e-05 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.305781+0000", 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.705217515999998, 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020476042999999999, 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.095 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305781+0000", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305781+0000", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305783+0000", 2024-09-16T02:15:04.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3970000000000001e-06 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305797+0000", 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4126e-05 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305882+0000", 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4503000000000001e-05 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305918+0000", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6140999999999998e-05 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324449+0000", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018531737999999999 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.326257+0000", 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001807138 2024-09-16T02:15:04.098 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.5s0 35/32 ECSubWriteReply(tid=3, last_complete=35'4, committed=1, applied=1))", 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.306734+0000", 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.704264332999998, 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019265915000000002, 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:04.099 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.306734+0000", 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.306734+0000", 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.100 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.306734+0000", 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.306739+0000", 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.023e-06 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.306986+0000", 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024619099999999999 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.307000+0000", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324770+0000", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017770454000000001 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.326000+0000", 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001229438 2024-09-16T02:15:04.102 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:73 3.5s0 3:a2a1f4c4:::smithi19530025-25:head [append 0~409600,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.329323+0000", 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.681675525999999, 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020279892000000001, 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.103 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 73 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.329323+0000", 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.329323+0000", 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.329326+0000", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9699999999999999e-06 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.330057+0000", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073137600000000005 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.330091+0000", 2024-09-16T02:15:04.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4601999999999998e-05 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.330128+0000", 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.6774999999999999e-05 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336641+0000", 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065130980000000002 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.106 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336902+0000", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026017100000000001 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.338703+0000", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001801591 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.349354+0000", 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010650498 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.349474+0000", 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120123 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.349602+0000", 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012868800000000001 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.108 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:255 3.5s0 3:a7785c58:::smithi19530025-8:head [setxattr _header (55) in=7b] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.390646+0000", 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.620352542999999, 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.28590590399999999, 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 255 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390646+0000", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390646+0000", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390647+0000", 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3489999999999999e-06 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390653+0000", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1700000000000002e-06 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390666+0000", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3087e-05 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390675+0000", 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6090000000000001e-06 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.396396+0000", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057210359999999997 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.396653+0000", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025678400000000003 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.397890+0000", 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0012373659999999999 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.112 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.399294+0000", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001403627 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.676393+0000", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27709919799999999 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.676551+0000", 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015867799999999999 2024-09-16T02:15:04.113 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:256 3.5s0 3:a7785c58:::smithi19530025-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.390690+0000", 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 29.620308418, 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.287677931, 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:04.114 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 256 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390690+0000", 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390690+0000", 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390691+0000", 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7300000000000005e-07 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390694+0000", 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9530000000000001e-06 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:04.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390706+0000", 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1086e-05 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.390716+0000", 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0773e-05 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.398551+0000", 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078350260000000001 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.398711+0000", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159388 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.677107+0000", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.27839625000000001 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.677224+0000", 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117339 2024-09-16T02:15:04.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.678368+0000", 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001143243 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:04.119 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:04.120 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:04.121 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:04.121 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:04.122 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:04.122 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:04.114+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:04.122 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:04.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:04.123 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:04.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:04.123 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:04.124 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:04.146 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:04.146 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:04.179 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:04.179 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:04.210 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:04.210 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:04.221 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:04.219+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:04.241 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:04.241 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:04.275 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:04.275 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:04.292 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:04.292 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:04.292 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:04.292 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:04.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:04.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:04.311 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:04.322 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:04.319+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:04.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:04.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:04.382 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:04.383 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:04.419 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:04.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:04.414+0000 7fdf1759e640 0 -- 172.21.15.3:0/3790372940 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7fdf18227ff0 msgr2=0x7fdf1822a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:04.422 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:04.419+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:04.429 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:04.429 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:04.444 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:04.523 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:04.521+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:04.566 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:04.566 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:04.581 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:04.624 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:04.621+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:04.715 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:04.715 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:04.725 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:04.721+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:04.729 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:04.826 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:04.823+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:04.858 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:04.858 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:04.860 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:04.926 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:04.922+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:04.974 INFO:tasks.rados.rados.0.smithi195.stdout:263: done (1 left) 2024-09-16T02:15:04.974 INFO:tasks.rados.rados.0.smithi195.stdout:265: done (0 left) 2024-09-16T02:15:04.975 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 14 current snap is 21 2024-09-16T02:15:04.975 INFO:tasks.rados.rados.0.smithi195.stdout:266: seq_num 110 ranges {0=688128,688128=507904,1196032=655360} 2024-09-16T02:15:04.979 INFO:tasks.rados.rados.0.smithi195.stdout:266: writing smithi19530025-14 from 0 to 688128 tid 1 2024-09-16T02:15:04.983 INFO:tasks.rados.rados.0.smithi195.stdout:266: writing smithi19530025-14 from 688128 to 1196032 tid 2 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:266: writing smithi19530025-14 from 1196032 to 1851392 tid 3 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:267: read oid 13 snap 20 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:267: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:268: read oid 33 snap 20 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:268: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:269: copy_from oid 46 from oid 41 current snap is 21 2024-09-16T02:15:04.988 INFO:tasks.rados.rados.0.smithi195.stdout:270: delete oid 4 current snap is 21 2024-09-16T02:15:05.014 INFO:tasks.rados.rados.0.smithi195.stdout:268: expect (ObjNum 32 snap 0 seq_num 32) 2024-09-16T02:15:05.014 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 170002 2024-09-16T02:15:05.020 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:05.027 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:05.022+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:05.045 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.046 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.048 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 484896 2024-09-16T02:15:05.049 INFO:tasks.rados.rados.0.smithi195.stdout:266: finishing write tid 1 to smithi19530025-14 2024-09-16T02:15:05.049 INFO:tasks.rados.rados.0.smithi195.stdout:266: finishing write tid 2 to smithi19530025-14 2024-09-16T02:15:05.049 INFO:tasks.rados.rados.0.smithi195.stdout:266: finishing write tid 3 to smithi19530025-14 2024-09-16T02:15:05.050 INFO:tasks.rados.rados.0.smithi195.stdout:267: expect (ObjNum 88 snap 12 seq_num 88) 2024-09-16T02:15:05.060 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:05.074 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 2516577 2024-09-16T02:15:05.078 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:05.117 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:05.128 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:05.122+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: finishing write tid 4 to smithi19530025-14 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: finishing write tid 5 to smithi19530025-14 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: oid 14 updating version 0 to 136 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: oid 14 updating version 136 to 138 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: oid 14 version 138 is already newer than 137 2024-09-16T02:15:05.152 INFO:tasks.rados.rados.0.smithi195.stdout:266: oid 14 updating version 138 to 139 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 14 v 139 (ObjNum 110 snap 21 seq_num 110) dirty exists 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:266: left oid 14 (ObjNum 110 snap 21 seq_num 110) 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:269: finishing copy_from racing read to smithi19530025-46 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:266: done (4 left) 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:267: done (3 left) 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:268: done (2 left) 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:270: done (1 left) 2024-09-16T02:15:05.153 INFO:tasks.rados.rados.0.smithi195.stdout:271: snap_remove snap 19 2024-09-16T02:15:05.178 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.178 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.187 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 11] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:15:05.188 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:05.188 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T02:15:05.188 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-16T02:15:05.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 0 2024-09-16T02:15:05.193 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:05.235 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:05.235+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:05.330 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:05.325+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:05.358 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.359 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.372 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:05.430 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:05.429+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:05.490 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.490 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.504 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:05.551 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:05.551+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:05.575 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:05.575 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:05.626 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.626 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.632 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:05.626+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:05.639 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:05.733 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:05.731+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:05.805 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.805 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:05.819 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:05.860 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:05.859+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:05.907 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-16T02:15:05.917 INFO:tasks.rados.rados.0.smithi195.stdout:271: done (1 left) 2024-09-16T02:15:05.917 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 7 current snap is 21 2024-09-16T02:15:05.917 INFO:tasks.rados.rados.0.smithi195.stdout:272: seq_num 112 ranges {1622016=720896,2342912=802816,3145728=360448} 2024-09-16T02:15:05.935 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:05.935+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:05.936 INFO:tasks.rados.rados.0.smithi195.stdout:272: writing smithi19530025-7 from 1622016 to 2342912 tid 1 2024-09-16T02:15:05.944 INFO:tasks.rados.rados.0.smithi195.stdout:272: writing smithi19530025-7 from 2342912 to 3145728 tid 2 2024-09-16T02:15:05.947 INFO:tasks.rados.rados.0.smithi195.stdout:272: writing smithi19530025-7 from 3145728 to 3506176 tid 3 2024-09-16T02:15:05.947 INFO:tasks.rados.rados.0.smithi195.stdout:273: snap_remove snap 16 2024-09-16T02:15:05.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:15:05.997 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:05.997 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.011 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:06.036 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:06.030+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:06.137 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:06.134+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:06.182 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.182 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.196 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:06.238 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:06.238+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:06.317 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.317 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:06.339 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:06.337+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:06.440 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:06.438+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:06.455 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.455 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:06.541 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:06.538+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:15:06.601 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.601 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:06.641 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:06.637+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:06.734 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.734 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.742 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:06.739+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:06.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:06.843 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:06.839+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:06.876 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:06.876 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:06.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:06.909 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:15:06.919 INFO:tasks.rados.rados.0.smithi195.stdout:273: done (2 left) 2024-09-16T02:15:06.919 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 49 current snap is 21 2024-09-16T02:15:06.919 INFO:tasks.rados.rados.0.smithi195.stdout:269: finishing copy_from to smithi19530025-46 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 46 v 69 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: finishing write tid 1 to smithi19530025-7 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: finishing write tid 2 to smithi19530025-7 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: finishing write tid 3 to smithi19530025-7 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: finishing write tid 4 to smithi19530025-7 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: finishing write tid 5 to smithi19530025-7 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: oid 7 updating version 0 to 34 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: oid 7 version 34 is already newer than 32 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: oid 7 version 34 is already newer than 33 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: oid 7 version 34 is already newer than 31 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 34 (ObjNum 112 snap 21 seq_num 112) dirty exists 2024-09-16T02:15:06.920 INFO:tasks.rados.rados.0.smithi195.stdout:272: left oid 7 (ObjNum 112 snap 21 seq_num 112) 2024-09-16T02:15:06.921 INFO:tasks.rados.rados.0.smithi195.stdout:274: seq_num 113 ranges {720896=737280,1458176=557056,2015232=606208} 2024-09-16T02:15:06.938 INFO:tasks.rados.rados.0.smithi195.stdout:274: writing smithi19530025-49 from 720896 to 1458176 tid 1 2024-09-16T02:15:06.946 INFO:tasks.rados.rados.0.smithi195.stdout:274: writing smithi19530025-49 from 1458176 to 2015232 tid 2 2024-09-16T02:15:06.953 INFO:tasks.rados.rados.0.smithi195.stdout:274: writing smithi19530025-49 from 2015232 to 2621440 tid 3 2024-09-16T02:15:06.953 INFO:tasks.rados.rados.0.smithi195.stdout:269: done (2 left) 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:272: done (1 left) 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:275: copy_from oid 37 from oid 11 current snap is 21 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:276: rollback oid 21 current snap is 21 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 21 to 11 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 35 current snap is 21 2024-09-16T02:15:06.954 INFO:tasks.rados.rados.0.smithi195.stdout:277: seq_num 114 ranges {0=753664,753664=688128,1441792=475136} 2024-09-16T02:15:06.963 INFO:tasks.rados.rados.0.smithi195.stdout:277: writing smithi19530025-35 from 0 to 753664 tid 1 2024-09-16T02:15:06.973 INFO:tasks.rados.rados.0.smithi195.stdout:277: writing smithi19530025-35 from 753664 to 1441792 tid 2 2024-09-16T02:15:06.979 INFO:tasks.rados.rados.0.smithi195.stdout:277: writing smithi19530025-35 from 1441792 to 1916928 tid 3 2024-09-16T02:15:06.979 INFO:tasks.rados.rados.0.smithi195.stdout:278: snap_remove snap 20 2024-09-16T02:15:06.982 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:06.978+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:07.032 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.032 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.044 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:07.041+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:07.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:07.145 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:07.142+0000 7f3497f53640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26347) UID: 0 2024-09-16T02:15:07.220 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.220 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:07.278 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:07.274+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:07.347 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:07.345+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:07.358 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.359 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:07.462 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:07.458+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:07.544 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.544 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.549 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:07.545+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:07.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:07.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:07.649 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:07.649+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:07.726 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.727 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:07.755 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:07.756 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:07.756 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:07.750+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:07.782 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:07.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:07.809 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:07.809 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:07.836 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:07.836 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:07.851 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:07.846+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:07.859 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:07.859 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:07.862 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:07.863 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:07.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:07.891 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:07.920 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:07.920 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:07.946 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:07.946 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:07.951 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:07.951+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:07.968 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:07.968 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:07.992 INFO:tasks.rados.rados.0.smithi195.stdout:278: done (4 left) 2024-09-16T02:15:07.992 INFO:tasks.rados.rados.0.smithi195.stdout:279: delete oid 14 current snap is 21 2024-09-16T02:15:07.993 INFO:tasks.rados.rados.0.smithi195.stdout:276: finishing rollback tid 0 to smithi19530025-21 2024-09-16T02:15:07.993 INFO:tasks.rados.rados.0.smithi195.stdout:276: finishing rollback tid 1 to smithi19530025-21 2024-09-16T02:15:07.993 INFO:tasks.rados.rados.0.smithi195.stdout:276: finishing rollback tid 2 to smithi19530025-21 2024-09-16T02:15:07.993 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 21 v 52 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:15:07.993 INFO:tasks.rados.rados.0.smithi195.stdout:274: finishing write tid 1 to smithi19530025-49 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:274: finishing write tid 2 to smithi19530025-49 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:274: finishing write tid 3 to smithi19530025-49 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:274: finishing write tid 4 to smithi19530025-49 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:275: finishing copy_from racing read to smithi19530025-37 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:275: finishing copy_from to smithi19530025-37 2024-09-16T02:15:07.994 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 78 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-09-16T02:15:07.995 INFO:tasks.rados.rados.0.smithi195.stdout:274: finishing write tid 5 to smithi19530025-49 2024-09-16T02:15:07.995 INFO:tasks.rados.rados.0.smithi195.stdout:274: oid 49 updating version 0 to 100 2024-09-16T02:15:07.995 INFO:tasks.rados.rados.0.smithi195.stdout:274: oid 49 version 100 is already newer than 97 2024-09-16T02:15:07.995 INFO:tasks.rados.rados.0.smithi195.stdout:274: oid 49 version 100 is already newer than 99 2024-09-16T02:15:07.996 INFO:tasks.rados.rados.0.smithi195.stdout:274: oid 49 version 100 is already newer than 98 2024-09-16T02:15:07.996 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 49 v 100 (ObjNum 113 snap 21 seq_num 113) dirty exists 2024-09-16T02:15:07.996 INFO:tasks.rados.rados.0.smithi195.stdout:274: left oid 49 (ObjNum 113 snap 21 seq_num 113) 2024-09-16T02:15:07.996 INFO:tasks.rados.rados.0.smithi195.stdout:277: finishing write tid 1 to smithi19530025-35 2024-09-16T02:15:07.996 INFO:tasks.rados.rados.0.smithi195.stdout:277: finishing write tid 2 to smithi19530025-35 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: finishing write tid 3 to smithi19530025-35 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: finishing write tid 4 to smithi19530025-35 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: finishing write tid 5 to smithi19530025-35 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: oid 35 updating version 0 to 102 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: oid 35 version 102 is already newer than 101 2024-09-16T02:15:07.997 INFO:tasks.rados.rados.0.smithi195.stdout:277: oid 35 updating version 102 to 104 2024-09-16T02:15:07.998 INFO:tasks.rados.rados.0.smithi195.stdout:277: oid 35 version 104 is already newer than 103 2024-09-16T02:15:07.998 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 35 v 104 (ObjNum 114 snap 21 seq_num 114) dirty exists 2024-09-16T02:15:07.998 INFO:tasks.rados.rados.0.smithi195.stdout:277: left oid 35 (ObjNum 114 snap 21 seq_num 114) 2024-09-16T02:15:08.000 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:08.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:08.004 INFO:tasks.rados.rados.0.smithi195.stdout:274: done (4 left) 2024-09-16T02:15:08.004 INFO:tasks.rados.rados.0.smithi195.stdout:275: done (3 left) 2024-09-16T02:15:08.004 INFO:tasks.rados.rados.0.smithi195.stdout:276: done (2 left) 2024-09-16T02:15:08.005 INFO:tasks.rados.rados.0.smithi195.stdout:277: done (1 left) 2024-09-16T02:15:08.005 INFO:tasks.rados.rados.0.smithi195.stdout:279: done (0 left) 2024-09-16T02:15:08.005 INFO:tasks.rados.rados.0.smithi195.stdout:280: rollback oid 44 current snap is 21 2024-09-16T02:15:08.005 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 44 to 11 2024-09-16T02:15:08.006 INFO:tasks.rados.rados.0.smithi195.stdout:281: delete oid 30 current snap is 21 2024-09-16T02:15:08.017 INFO:tasks.rados.rados.0.smithi195.stdout:281: done (1 left) 2024-09-16T02:15:08.017 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 29 current snap is 21 2024-09-16T02:15:08.017 INFO:tasks.rados.rados.0.smithi195.stdout:282: seq_num 117 ranges {475136=425984,901120=606208,1507328=540672,2048000=409600} 2024-09-16T02:15:08.020 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:08.021 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:08.022 INFO:tasks.rados.rados.0.smithi195.stdout:282: writing smithi19530025-29 from 475136 to 901120 tid 1 2024-09-16T02:15:08.026 INFO:tasks.rados.rados.0.smithi195.stdout:282: writing smithi19530025-29 from 901120 to 1507328 tid 2 2024-09-16T02:15:08.029 INFO:tasks.rados.rados.0.smithi195.stdout:282: writing smithi19530025-29 from 1507328 to 2048000 tid 3 2024-09-16T02:15:08.034 INFO:tasks.rados.rados.0.smithi195.stdout:282: writing smithi19530025-29 from 2048000 to 2457600 tid 4 2024-09-16T02:15:08.034 INFO:tasks.rados.rados.0.smithi195.stdout:283: read oid 5 snap -1 2024-09-16T02:15:08.034 INFO:tasks.rados.rados.0.smithi195.stdout:283: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:15:08.034 INFO:tasks.rados.rados.0.smithi195.stdout:284: read oid 37 snap -1 2024-09-16T02:15:08.034 INFO:tasks.rados.rados.0.smithi195.stdout:284: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:285: read oid 50 snap 11 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:285: expect (ObjNum 71 snap 7 seq_num 71) 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:286: copy_from oid 3 from oid 24 current snap is 21 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:287: snap_create 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:280: finishing rollback tid 0 to smithi19530025-44 2024-09-16T02:15:08.035 INFO:tasks.rados.rados.0.smithi195.stdout:280: finishing rollback tid 1 to smithi19530025-44 2024-09-16T02:15:08.036 INFO:tasks.rados.rados.0.smithi195.stdout:280: finishing rollback tid 2 to smithi19530025-44 2024-09-16T02:15:08.036 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 148 (ObjNum 57 snap 5 seq_num 57) dirty exists 2024-09-16T02:15:08.036 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 1 to smithi19530025-29 2024-09-16T02:15:08.037 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:08.038 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:08.038 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 2 to smithi19530025-29 2024-09-16T02:15:08.038 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 3 to smithi19530025-29 2024-09-16T02:15:08.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:08.041 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 4 to smithi19530025-29 2024-09-16T02:15:08.046 INFO:tasks.rados.rados.0.smithi195.stdout:285: expect (ObjNum 71 snap 7 seq_num 71) 2024-09-16T02:15:08.051 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 365291 2024-09-16T02:15:08.061 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:08.062 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:08.062 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:08.054+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:08.083 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:08.083 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:08.086 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:08.109 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:08.109 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:08.117 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:08.130+0000 7fb69effd640 0 -- 172.21.15.3:0/4088037511 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7fb6a02244f0 msgr2=0x7fb6a02268e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:08.141 INFO:tasks.rados.rados.0.smithi195.stdout:284: expect (ObjNum 76 snap 8 seq_num 76) 2024-09-16T02:15:08.141 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:08.144 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:08.153 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:08.153+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:08.165 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 947105 2024-09-16T02:15:08.173 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 328282 2024-09-16T02:15:08.174 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:08.180 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:08.180 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:08.180 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:08.189 INFO:tasks.rados.rados.0.smithi195.stdout:283: expect (ObjNum 84 snap 10 seq_num 84) 2024-09-16T02:15:08.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:08.197 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1336293 2024-09-16T02:15:08.198 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:08.212 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 165391 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:286: finishing copy_from racing read to smithi19530025-3 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 5 to smithi19530025-29 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: finishing write tid 6 to smithi19530025-29 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: oid 29 updating version 0 to 42 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: oid 29 version 42 is already newer than 40 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: oid 29 updating version 42 to 44 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: oid 29 version 44 is already newer than 43 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:282: oid 29 version 44 is already newer than 41 2024-09-16T02:15:08.223 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 29 v 44 (ObjNum 117 snap 21 seq_num 117) dirty exists 2024-09-16T02:15:08.224 INFO:tasks.rados.rados.0.smithi195.stdout:282: left oid 29 (ObjNum 117 snap 21 seq_num 117) 2024-09-16T02:15:08.224 INFO:tasks.rados.rados.0.smithi195.stdout:286: finishing copy_from to smithi19530025-3 2024-09-16T02:15:08.224 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 3 v 21 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-16T02:15:08.282 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:08.278+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:08.355 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:08.350+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:08.364 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:08.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:46 3.0s0 3:0952f51a:::smithi19530025-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 44.060555815000001, 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013865147, 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.365 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300541+0000", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.193e-06 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301465+0000", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092351300000000005 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301478+0000", 2024-09-16T02:15:08.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.348e-05 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301487+0000", 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3869999999999995e-06 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301559+0000", 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2444e-05 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301744+0000", 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185019 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.302843+0000", 2024-09-16T02:15:08.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010987550000000001 2024-09-16T02:15:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305418+0000", 2024-09-16T02:15:08.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025746219999999999 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314319+0000", 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089012259999999999 2024-09-16T02:15:08.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314404+0000", 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5507999999999995e-05 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:47 3.0s0 3:0952f51a:::smithi19530025-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 44.059587555, 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013156407, 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 47 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:08.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301508+0000", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079e-06 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301513+0000", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.194e-06 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301521+0000", 2024-09-16T02:15:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2309999999999996e-06 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301527+0000", 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4710000000000004e-06 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304040+0000", 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025124589999999999 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304279+0000", 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023908499999999999 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304869+0000", 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058981500000000004 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.307149+0000", 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022805849999999999 2024-09-16T02:15:08.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314589+0000", 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074399310000000003 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314664+0000", 2024-09-16T02:15:08.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:48 3.0s0 3:0952f51a:::smithi19530025-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 44.059401772000001, 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021785207000000001, 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.379 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 48 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0359999999999999e-06 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301701+0000", 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9700000000000002e-06 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301718+0000", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7201999999999999e-05 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301729+0000", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0516999999999999e-05 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305321+0000", 2024-09-16T02:15:08.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035922570000000002 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305405+0000", 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3998000000000005e-05 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314939+0000", 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095341579999999992 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.315042+0000", 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102858 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323478+0000", 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084362110000000008 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.384 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:139 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~786432,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.944453377999999, 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066880840000000004, 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 139 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416642+0000", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.416644+0000", 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9860000000000001e-06 2024-09-16T02:15:08.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417552+0000", 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090839999999999996 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417565+0000", 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3406e-05 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417572+0000", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2779999999999999e-06 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417622+0000", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0717999999999998e-05 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417725+0000", 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102357 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.418463+0000", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073846399999999998 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419987+0000", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001523735 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423245+0000", 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032577930000000001 2024-09-16T02:15:08.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423330+0000", 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4946999999999998e-05 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:141 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.943489892000002, 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073674559999999997, 2024-09-16T02:15:08.390 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417605+0000", 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417608+0000", 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8899999999999999e-06 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417612+0000", 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0770000000000001e-06 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.417616+0000", 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1389999999999999e-06 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419930+0000", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023145040000000002 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.419980+0000", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0439000000000003e-05 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.423911+0000", 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039302770000000002 2024-09-16T02:15:08.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424016+0000", 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104734 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.424973+0000", 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095702600000000004 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:252 3.0s0 3:0952f51a:::smithi19530025-16:head [stat,delete] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.016167916999997, 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011932368, 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 252 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344929+0000", 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2709999999999999e-06 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344944+0000", 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-05 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344959+0000", 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4990999999999999e-05 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344967+0000", 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9670000000000001e-06 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345036+0000", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9412999999999994e-05 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345046+0000", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6390000000000004e-06 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346669+0000", 2024-09-16T02:15:08.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016225289999999999 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346810+0000", 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141457 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347909+0000", 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010991670000000001 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349212+0000", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013032880000000001 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356764+0000", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007551978 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4977999999999998e-05 2024-09-16T02:15:08.402 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.009927998999999, 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095540220000000006, 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351168+0000", 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7899999999999997e-07 2024-09-16T02:15:08.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351191+0000", 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3377e-05 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351238+0000", 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351247+0000", 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2310000000000002e-06 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351304+0000", 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7058000000000002e-05 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351312+0000", 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8790000000000003e-06 2024-09-16T02:15:08.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360676+0000", 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093640279999999999 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360721+0000", 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4975999999999998e-05 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.003493061999997, 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066953009999999999, 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357602+0000", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357623+0000", 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0469999999999999e-05 2024-09-16T02:15:08.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357662+0000", 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9601999999999998e-05 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357670+0000", 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8320000000000006e-06 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357735+0000", 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4970999999999998e-05 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357744+0000", 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3899999999999999e-06 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364244+0000", 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064994440000000001 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364297+0000", 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3393999999999997e-05 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs3 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:15:08.412 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.002818423000001, 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081662019999999991, 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358277+0000", 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5699999999999998e-07 2024-09-16T02:15:08.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358303+0000", 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6217999999999999e-05 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358461+0000", 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015794100000000001 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358471+0000", 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5030000000000003e-06 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358566+0000", 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5378999999999996e-05 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.358578+0000", 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.254e-05 2024-09-16T02:15:08.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366371+0000", 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077921409999999998 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366443+0000", 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2123e-05 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 45/32 ECSubWriteReply(tid=12, last_complete=45'3, committed=1, applied=1))", 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.000142652999997, 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82653599300000002, 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:08.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360954+0000", 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42e-06 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360960+0000", 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6340000000000003e-06 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187311+0000", 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82635024400000001 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187328+0000", 2024-09-16T02:15:08.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7880000000000002e-05 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187449+0000", 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012031000000000001 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187488+0000", 2024-09-16T02:15:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9505000000000002e-05 2024-09-16T02:15:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:15:08.423 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 33.998774601999997, 2024-09-16T02:15:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074648689999999998, 2024-09-16T02:15:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:15:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362320+0000", 2024-09-16T02:15:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362321+0000", 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8199999999999999e-07 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362342+0000", 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0996999999999999e-05 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362436+0000", 2024-09-16T02:15:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3445999999999999e-05 2024-09-16T02:15:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362445+0000", 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3950000000000007e-06 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362539+0000", 2024-09-16T02:15:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3527000000000001e-05 2024-09-16T02:15:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362548+0000", 2024-09-16T02:15:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9429999999999995e-06 2024-09-16T02:15:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369727+0000", 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071787120000000003 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369785+0000", 2024-09-16T02:15:08.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8066999999999998e-05 2024-09-16T02:15:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:404 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~606208] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.398910349000001, 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022657539000000001, 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 404 2024-09-16T02:15:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:08.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962187+0000", 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8729999999999999e-06 2024-09-16T02:15:08.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963628+0000", 2024-09-16T02:15:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001440113 2024-09-16T02:15:08.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963678+0000", 2024-09-16T02:15:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0349999999999997e-05 2024-09-16T02:15:08.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963702+0000", 2024-09-16T02:15:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4533e-05 2024-09-16T02:15:08.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963873+0000", 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170669 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.964084+0000", 2024-09-16T02:15:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021121200000000001 2024-09-16T02:15:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966544+0000", 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024596430000000001 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.969112+0000", 2024-09-16T02:15:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025675950000000002 2024-09-16T02:15:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984664+0000", 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015552145999999999 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984842+0000", 2024-09-16T02:15:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178405 2024-09-16T02:15:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:405 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~720896] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.386595923000002, 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011549382, 2024-09-16T02:15:08.449 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.451 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 405 2024-09-16T02:15:08.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.451 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:08.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974502+0000", 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975490+0000", 2024-09-16T02:15:08.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098815699999999992 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975523+0000", 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3583e-05 2024-09-16T02:15:08.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975539+0000", 2024-09-16T02:15:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5330000000000001e-05 2024-09-16T02:15:08.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975692+0000", 2024-09-16T02:15:08.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015385600000000001 2024-09-16T02:15:08.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975935+0000", 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024296100000000001 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977706+0000", 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017700909999999999 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984097+0000", 2024-09-16T02:15:08.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063910879999999996 2024-09-16T02:15:08.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985929+0000", 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018325940000000001 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986048+0000", 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119226 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:406 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~49152] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.385522483999999, 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011213219999999999, 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.462 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 406 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:08.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975574+0000", 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4300000000000001e-06 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975631+0000", 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7519999999999998e-05 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975655+0000", 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3465999999999999e-05 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975671+0000", 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6186999999999999e-05 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.979950+0000", 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042792400000000001 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.980113+0000", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016220199999999999 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981185+0000", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001072092 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984325+0000", 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031403780000000001 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986680+0000", 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023551359999999999 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986786+0000", 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010556900000000001 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.467 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:407 3.9s0 3:9a381d46:::smithi19530025-47:head [setxattr _header (57) in=7b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.385394184999999, 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.35746389899999997, 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 407 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975702+0000", 2024-09-16T02:15:08.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.57e-06 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975711+0000", 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2900000000000002e-06 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975730+0000", 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9017000000000001e-05 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975749+0000", 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8850000000000001e-05 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982236+0000", 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064873109999999999 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982511+0000", 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027516800000000001 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.983460+0000", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094918300000000004 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985743+0000", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00228254 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.332957+0000", 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.347213828 2024-09-16T02:15:08.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333165+0000", 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000208142 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:408 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.385324269000002, 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.36004730099999999, 2024-09-16T02:15:08.473 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 408 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975772+0000", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.544e-06 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975779+0000", 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2829999999999998e-06 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975801+0000", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2082999999999998e-05 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975816+0000", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5662999999999999e-05 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986954+0000", 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011137461 2024-09-16T02:15:08.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.987093+0000", 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139436 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333971+0000", 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34687802400000001 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:08.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.334106+0000", 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013453200000000001 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.335818+0000", 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017122750000000001 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:15:08.478 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 21.398023229, 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069953569999999998, 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:15:08.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963072+0000", 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3000000000000001e-07 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963092+0000", 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9803e-05 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963251+0000", 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015900899999999999 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.963269+0000", 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8363e-05 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.965522+0000", 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022524340000000002 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.965546+0000", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4485999999999999e-05 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.969996+0000", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044494419999999996 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.970067+0000", 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1390000000000006e-05 2024-09-16T02:15:08.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds3 73/32 ECSubWrite(tid=87, reqid=osd.6.0:87, at_version=73'72, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.948063+0000", 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.4130317330000004, 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082543540000000002, 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948063+0000", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948063+0000", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948064+0000", 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079e-06 2024-09-16T02:15:08.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948091+0000", 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6945e-05 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948147+0000", 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5964000000000002e-05 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948158+0000", 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0934999999999999e-05 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948274+0000", 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115482 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948301+0000", 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7773e-05 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.956226+0000", 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079242440000000004 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.956318+0000", 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1932e-05 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s6 73/54 ECSubWrite(tid=101, reqid=osd.7.0:101, at_version=73'16, trim_to=0'0, roll_forward_to=70'14))", 2024-09-16T02:15:08.487 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.950002+0000", 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.4110926609999996, 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080915029999999999, 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950002+0000", 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950002+0000", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950003+0000", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2399999999999999e-07 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950030+0000", 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7233000000000002e-05 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950086+0000", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5899000000000001e-05 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950096+0000", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0158e-05 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950247+0000", 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015145499999999999 2024-09-16T02:15:08.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.950275+0000", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7596000000000001e-05 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958008+0000", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077333749999999998 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958094+0000", 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5562999999999994e-05 2024-09-16T02:15:08.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s1 74/32 ECSubWrite(tid=277, reqid=osd.13.0:277, at_version=74'88, trim_to=0'0, roll_forward_to=73'84))", 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.4008768749999998, 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081637649999999999, 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960219+0000", 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9999999999999996e-07 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960264+0000", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5043000000000002e-05 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960498+0000", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000234408 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.960516+0000", 2024-09-16T02:15:08.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7790000000000001e-05 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961741+0000", 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001224901 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961766+0000", 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5304e-05 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968278+0000", 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006511363 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968382+0000", 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104156 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:08.496 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:08.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:08.498 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:08.495+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:08.599 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:08.599+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:08.633 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:08.633 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:08.633 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:08.633 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:08.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:08.700 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:08.699+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:08.771 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:08.771 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:08.771 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:08.771 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:08.771 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:08.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:08.800 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:08.798+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:08.901 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:08.899+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:192, at_version=38'10, trim_to=0'0, roll_forward_to=35'8))", 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 41.590652011000003, 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051833920000000002, 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.910 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318272+0000", 2024-09-16T02:15:08.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318273+0000", 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4999999999999998e-07 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318481+0000", 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020832199999999999 2024-09-16T02:15:08.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318589+0000", 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107645 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318612+0000", 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3468999999999998e-05 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318803+0000", 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019100200000000001 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318832+0000", 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8812000000000001e-05 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323396+0000", 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045639950000000004 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.323455+0000", 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9697000000000001e-05 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 45/32 ECSubWrite(tid=17, reqid=osd.8.0:17, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.545954797, 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056123190000000002, 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.915 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362970+0000", 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.44e-07 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363028+0000", 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7784000000000001e-05 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364099+0000", 2024-09-16T02:15:08.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00107157 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364121+0000", 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2206000000000001e-05 2024-09-16T02:15:08.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364291+0000", 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017008400000000001 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364317+0000", 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5596e-05 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368520+0000", 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042029670000000002 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368582+0000", 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1568000000000004e-05 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.923 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=29, reqid=osd.7.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=45'10))", 2024-09-16T02:15:08.923 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:08.923 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 34.545795372000001, 2024-09-16T02:15:08.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053661100000000003, 2024-09-16T02:15:08.923 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363130+0000", 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5600000000000004e-07 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363151+0000", 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1073e-05 2024-09-16T02:15:08.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364189+0000", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010384350000000001 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364205+0000", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5732000000000001e-05 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364314+0000", 2024-09-16T02:15:08.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010873400000000001 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364328+0000", 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4525000000000001e-05 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368397+0000", 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040685349999999999 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368495+0000", 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8120000000000002e-05 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s2 52/32 ECSubWrite(tid=25, reqid=client.4598.0:397, at_version=52'15, trim_to=0'0, roll_forward_to=51'14))", 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.983558084999999, 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048112190000000003, 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.928 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:15:08.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925366+0000", 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925367+0000", 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3600000000000002e-07 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925676+0000", 2024-09-16T02:15:08.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030898500000000001 2024-09-16T02:15:08.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925831+0000", 2024-09-16T02:15:08.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015518199999999999 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.925908+0000", 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6587000000000004e-05 2024-09-16T02:15:08.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.926247+0000", 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033902999999999998 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.926275+0000", 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7926999999999999e-05 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.930095+0000", 2024-09-16T02:15:08.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038202650000000002 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.930177+0000", 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2406999999999999e-05 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=22, reqid=client.4598.0:404, at_version=52'7, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:08.936 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.941294586000001, 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013916968, 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967631+0000", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967931+0000", 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030039800000000001 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979030+0000", 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011099121999999999 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979051+0000", 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0577e-05 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979192+0000", 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140526 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979207+0000", 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981459+0000", 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022516110000000002 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981546+0000", 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7508000000000003e-05 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=23, reqid=client.4598.0:405, at_version=52'8, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.930301790000001, 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050943480000000003, 2024-09-16T02:15:08.941 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978622+0000", 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978624+0000", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.978992+0000", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036885999999999999 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982073+0000", 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030808860000000001 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982089+0000", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5310000000000001e-05 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982188+0000", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9246999999999994e-05 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.982203+0000", 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4600000000000001e-05 2024-09-16T02:15:08.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.983631+0000", 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014287810000000001 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.983717+0000", 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5371000000000006e-05 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=25, reqid=client.4598.0:407, at_version=52'10, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.924711599999998, 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34897630299999999, 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984214+0000", 2024-09-16T02:15:08.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.384e-06 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984244+0000", 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0549000000000002e-05 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331741+0000", 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34749696499999999 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331756+0000", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4873e-05 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331921+0000", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016467900000000001 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331952+0000", 2024-09-16T02:15:08.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0895000000000002e-05 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333108+0000", 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001156633 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333189+0000", 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0325000000000001e-05 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.950 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9531722990000002, 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060351959999999996, 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955857+0000", 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010480999999999999 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955998+0000", 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014078300000000001 2024-09-16T02:15:08.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956026+0000", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8285000000000001e-05 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956200+0000", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174165 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956232+0000", 2024-09-16T02:15:08.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1047000000000003e-05 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961698+0000", 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054662360000000002 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961787+0000", 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9234000000000007e-05 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9492992649999996, 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060372860000000002, 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.956 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959627+0000", 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9099999999999999e-06 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959685+0000", 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8099000000000001e-05 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959791+0000", 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106409 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959814+0000", 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3224999999999999e-05 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960393+0000", 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057815700000000004 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960417+0000", 2024-09-16T02:15:08.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4808e-05 2024-09-16T02:15:08.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965585+0000", 2024-09-16T02:15:08.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051671030000000001 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965662+0000", 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7575000000000002e-05 2024-09-16T02:15:08.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/32 ECSubWrite(tid=277, reqid=osd.13.0:277, at_version=74'88, trim_to=0'0, roll_forward_to=73'84))", 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:08.964 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9482652080000005, 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057458700000000001, 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:08.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960660+0000", 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0400000000000005e-07 2024-09-16T02:15:08.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960691+0000", 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1438e-05 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960918+0000", 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227304 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960949+0000", 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0141000000000002e-05 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962083+0000", 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011340129999999999 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962117+0000", 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3921999999999998e-05 2024-09-16T02:15:08.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966316+0000", 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041993250000000003 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966405+0000", 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8923000000000002e-05 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.963176+0000", 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9457483839999998, 2024-09-16T02:15:08.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049299569999999996, 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963176+0000", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963176+0000", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963178+0000", 2024-09-16T02:15:08.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963228+0000", 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0705999999999999e-05 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963333+0000", 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104413 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963355+0000", 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2147999999999999e-05 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963567+0000", 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000211735 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.963593+0000", 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5987999999999999e-05 2024-09-16T02:15:08.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968016+0000", 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044231130000000002 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968106+0000", 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9963999999999994e-05 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 74/32 ECSubWrite(tid=114, reqid=osd.7.0:114, at_version=74'29, trim_to=0'0, roll_forward_to=73'27))", 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9434052790000003, 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085142380000000004, 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965520+0000", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4e-07 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965569+0000", 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9175999999999998e-05 2024-09-16T02:15:08.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965669+0000", 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9946999999999997e-05 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965692+0000", 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3422e-05 2024-09-16T02:15:08.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965858+0000", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016617800000000001 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965887+0000", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8781000000000001e-05 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973464+0000", 2024-09-16T02:15:08.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075773079999999996 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974033+0000", 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056858600000000005 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=112, reqid=client.4598.0:655, at_version=74'82, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9402459910000003, 2024-09-16T02:15:08.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062268389999999996, 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968680+0000", 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.903e-06 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968727+0000", 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7139999999999999e-05 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968819+0000", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1711000000000003e-05 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968841+0000", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2314e-05 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:08.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000169872 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969038+0000", 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6755e-05 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974822+0000", 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005783697 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974905+0000", 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3447000000000003e-05 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/32 ECSubWrite(tid=278, reqid=osd.13.0:277, at_version=0'0, trim_to=0'0, roll_forward_to=74'88))", 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.970843+0000", 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9380813259999998, 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048810529999999998, 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.987 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970843+0000", 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970843+0000", 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970845+0000", 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8199999999999999e-06 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970875+0000", 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0443999999999999e-05 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970966+0000", 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1176999999999998e-05 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.970990+0000", 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3708999999999999e-05 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.971147+0000", 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000157202 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.971174+0000", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7365e-05 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.975634+0000", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044595930000000004 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.975724+0000", 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9742999999999998e-05 2024-09-16T02:15:08.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 74/66 ECSubWrite(tid=68, reqid=client.4598.0:653, at_version=74'44, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9358053929999999, 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069659270000000002, 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.992 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973120+0000", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7399999999999991e-07 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973262+0000", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000142483 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973686+0000", 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042359299999999999 2024-09-16T02:15:08.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973721+0000", 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5491000000000003e-05 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973886+0000", 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164372 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:08.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974322+0000", 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000436556 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.979991+0000", 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056687619999999999 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980085+0000", 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3696000000000006e-05 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.996 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=113, reqid=client.4598.0:656, at_version=74'83, trim_to=0'0, roll_forward_to=74'77))", 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.973640+0000", 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9352844329999996, 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052357660000000002, 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973640+0000", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973640+0000", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973642+0000", 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8870000000000001e-06 2024-09-16T02:15:08.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973702+0000", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0569000000000002e-05 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973807+0000", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010471600000000001 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973829+0000", 2024-09-16T02:15:08.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2544999999999998e-05 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974041+0000", 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021209199999999999 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974068+0000", 2024-09-16T02:15:09.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6730999999999999e-05 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978785+0000", 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004716362 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978875+0000", 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0864000000000003e-05 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.001 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 74/66 ECSubWrite(tid=69, reqid=osd.8.0:69, at_version=74'46, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.978535+0000", 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.9303887199999998, 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004938876, 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978535+0000", 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978535+0000", 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978537+0000", 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.666e-06 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978611+0000", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3662e-05 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978710+0000", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9306000000000001e-05 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.978734+0000", 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3719000000000001e-05 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.979087+0000", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000353288 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.979114+0000", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7039e-05 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.983359+0000", 2024-09-16T02:15:09.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042448870000000001 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.983474+0000", 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115309 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 75/66 ECSubWrite(tid=119, reqid=client.4598.0:660, at_version=75'42, trim_to=0'0, roll_forward_to=74'40))", 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9472362409999997, 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053709489999999999, 2024-09-16T02:15:09.006 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9299999999999999e-07 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961729+0000", 2024-09-16T02:15:09.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0330999999999997e-05 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961826+0000", 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7479999999999995e-05 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961855+0000", 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8792000000000001e-05 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.962022+0000", 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016759199999999999 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.962055+0000", 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2801e-05 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.966964+0000", 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00490859 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.967059+0000", 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4969999999999994e-05 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 79/32 ECSubWrite(tid=138, reqid=osd.7.0:138, at_version=79'38, trim_to=0'0, roll_forward_to=78'36))", 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.918668979, 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079362210000000002, 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.011 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990256+0000", 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990299+0000", 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2870999999999997e-05 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990371+0000", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2208999999999999e-05 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990390+0000", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8802000000000001e-05 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990544+0000", 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154661 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990576+0000", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1655000000000001e-05 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998130+0000", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075539659999999996 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998191+0000", 2024-09-16T02:15:09.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1407000000000003e-05 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.91723063900000001, 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068584240000000001, 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991694+0000", 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-16T02:15:09.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991735+0000", 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1068000000000001e-05 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991880+0000", 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014438400000000001 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991902+0000", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2616000000000002e-05 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992060+0000", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015784600000000001 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992086+0000", 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6239000000000001e-05 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998446+0000", 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063592730000000004 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998552+0000", 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106121 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.021 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:15:09.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:09.023 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:09.018+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:09.123 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:09.118+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:09.150 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.151 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:09.151 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:09.151 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:09.224 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:09.221+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:09.287 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.287 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:09.287 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:09.287 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:09.287 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:09.324 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:09.322+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:09.425 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:09.418+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:09.471 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:192 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~557056] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.153920917000001, 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015517632999999999, 2024-09-16T02:15:09.472 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 192 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314812+0000", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029999999999999e-06 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315788+0000", 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097551599999999995 2024-09-16T02:15:09.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315825+0000", 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6742000000000002e-05 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315847+0000", 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2396e-05 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316018+0000", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170729 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316179+0000", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016178000000000001 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318923+0000", 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002743381 2024-09-16T02:15:09.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071732469999999998 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330202+0000", 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004106391 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330327+0000", 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012444800000000001 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:194 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (56) in=7b] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.146741892999998, 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011982552, 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.478 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 194 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321988+0000", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.321990+0000", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.756e-06 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322010+0000", 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9443e-05 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322033+0000", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322049+0000", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6201999999999998e-05 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326191+0000", 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041419430000000004 2024-09-16T02:15:09.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326397+0000", 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020605800000000001 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327141+0000", 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074450000000000004 2024-09-16T02:15:09.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.328976+0000", 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018343109999999999 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333836+0000", 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048599389999999998 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.333971+0000", 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013532799999999999 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:195 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.146647407000003, 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014685304, 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.484 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 195 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322084+0000", 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322092+0000", 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6529999999999994e-06 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.486 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322111+0000", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8909000000000001e-05 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322124+0000", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3589e-05 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327750+0000", 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056258819999999996 2024-09-16T02:15:09.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327883+0000", 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013255499999999999 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334789+0000", 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069061050000000001 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335013+0000", 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022373400000000001 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.336768+0000", 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001755454 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.489 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:243 3.8s0 3:165ce5f6:::smithi19530025-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.133645385000001, 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01238621, 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335085+0000", 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335086+0000", 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.66e-06 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335094+0000", 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9829999999999996e-06 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335118+0000", 2024-09-16T02:15:09.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3716000000000001e-05 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.335135+0000", 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7102999999999999e-05 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336819+0000", 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001684051 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.336943+0000", 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012403199999999999 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.346988+0000", 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010044599 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347205+0000", 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021740900000000001 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.347471+0000", 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026565699999999998 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.495 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:504 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~786432] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.472701453999999, 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022327653999999999, 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.496 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 504 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996033+0000", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1969999999999998e-06 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.997993+0000", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019604309999999999 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998029+0000", 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5525999999999998e-05 2024-09-16T02:15:09.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998044+0000", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5440000000000001e-05 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998168+0000", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012395 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998340+0000", 2024-09-16T02:15:09.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171619 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.000528+0000", 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021878459999999998 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004445+0000", 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003916966 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018230+0000", 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013784892 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018356+0000", 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126787 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.501 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:505 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~491520] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.465359769999999, 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022250066999999998, 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.502 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 505 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003374+0000", 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.517e-06 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004387+0000", 2024-09-16T02:15:09.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001013445 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004415+0000", 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004434+0000", 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9966000000000002e-05 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004556+0000", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121734 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004705+0000", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014862999999999999 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.006184+0000", 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014795570000000001 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.008703+0000", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025187090000000001 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025506+0000", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016802766 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025620+0000", 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114627 2024-09-16T02:15:09.508 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:506 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~278528] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.458849031, 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016078973, 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.509 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 506 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009892+0000", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0849999999999999e-05 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010263+0000", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037122700000000002 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010289+0000", 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5979e-05 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010303+0000", 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3876999999999999e-05 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010423+0000", 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119824 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010635+0000", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000212091 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.011728+0000", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001092927 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015127+0000", 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033989139999999998 2024-09-16T02:15:09.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025851+0000", 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01072415 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025960+0000", 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109134 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:507 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (57) in=7b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.458395359000001, 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016008728999999999, 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 507 2024-09-16T02:15:09.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010336+0000", 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.623e-06 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010345+0000", 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4500000000000004e-06 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010364+0000", 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010379+0000", 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.012892+0000", 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002513286 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.013156+0000", 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026390700000000002 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014239+0000", 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00108344 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015432+0000", 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192961 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026211+0000", 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010778424999999999 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026344+0000", 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013285099999999999 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:508 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.458329620000001, 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019269792000000001, 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 508 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010402+0000", 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:15:09.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010408+0000", 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7699999999999998e-06 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010425+0000", 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7603000000000001e-05 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010443+0000", 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8202999999999998e-05 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014784+0000", 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043404239999999998 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014911+0000", 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012705400000000001 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027153+0000", 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012242116000000001 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027274+0000", 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012078099999999999 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.029670+0000", 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023965150000000001 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:586 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.559571004, 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018679077999999998, 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.526 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 586 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909162+0000", 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.897e-06 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909177+0000", 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4630000000000001e-05 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909215+0000", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8588999999999999e-05 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909237+0000", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.211e-05 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909418+0000", 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000181003 2024-09-16T02:15:09.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909440+0000", 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1271999999999999e-05 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.915451+0000", 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006010858 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.530 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.917204+0000", 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017530720000000001 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.920089+0000", 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002885267 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926620+0000", 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065311190000000002 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927778+0000", 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001157571 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927838+0000", 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0690000000000003e-05 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:587 3.8s0 3:165ce5f6:::smithi19530025-40:head [rollback d] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:09.532 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.559451442, 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023594436999999999, 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 587 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909280+0000", 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-16T02:15:09.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8129999999999998e-06 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909310+0000", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2731999999999999e-05 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909329+0000", 2024-09-16T02:15:09.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9134e-05 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909687+0000", 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035797500000000001 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909698+0000", 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.536 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921510+0000", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011812049999999999 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921788+0000", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027776500000000002 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.924415+0000", 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00262737 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926686+0000", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022710270000000001 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932811+0000", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061244309999999996 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932873+0000", 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2630000000000002e-05 2024-09-16T02:15:09.538 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:588 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.559347098, 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026094544000000001, 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.539 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 588 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:09.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909384+0000", 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909391+0000", 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6159999999999998e-06 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909409+0000", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8335999999999999e-05 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909424+0000", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.448e-05 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909774+0000", 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035054599999999997 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909792+0000", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.77e-05 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925523+0000", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015730929000000001 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925651+0000", 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012825900000000001 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926311+0000", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065990199999999997 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927500+0000", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001188599 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935426+0000", 2024-09-16T02:15:09.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079261060000000005 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935478+0000", 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1684000000000002e-05 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:594 3.8s0 3:1e2b38c5:::smithi19530025-2:head [append 0~524288] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:09.545 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.537705143, 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013176393, 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 594 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:09.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931025+0000", 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931028+0000", 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6170000000000001e-06 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931898+0000", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086988199999999999 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931932+0000", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3949000000000003e-05 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.931948+0000", 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6353000000000001e-05 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.932115+0000", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000167207 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.932417+0000", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030156999999999997 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.935157+0000", 2024-09-16T02:15:09.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027404019999999999 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.943666+0000", 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085089740000000007 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.943973+0000", 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030676400000000001 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.944201+0000", 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022867499999999999 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:596 3.8s0 3:1e2b38c5:::smithi19530025-2:head [append 0~409600] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.529067218, 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014017419999999999, 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.551 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 596 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939663+0000", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.939666+0000", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.96e-06 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940429+0000", 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00076330500000000004 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940458+0000", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8373000000000001e-05 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940477+0000", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9352999999999999e-05 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.945478+0000", 2024-09-16T02:15:09.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005001325 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.945625+0000", 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014625 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.947542+0000", 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019176740000000001 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.951638+0000", 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040957040000000004 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.953483+0000", 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018450929999999999 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.953680+0000", 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000197383 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:597 3.8s0 3:1e2b38c5:::smithi19530025-2:head [setxattr _header (56) in=7b] snapc 14=[14,13,12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.528225098, 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014074267999999999, 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 597 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940505+0000", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940506+0000", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1829999999999999e-06 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940514+0000", 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4399999999999999e-06 2024-09-16T02:15:09.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940525+0000", 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1537e-05 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940534+0000", 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9230000000000008e-06 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.948943+0000", 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084088259999999995 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.949213+0000", 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026987099999999998 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.950685+0000", 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014722870000000001 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.952884+0000", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021985749999999999 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.954457+0000", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001572988 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.954579+0000", 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000122638 2024-09-16T02:15:09.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:598 3.8s0 3:1e2b38c5:::smithi19530025-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.528182996, 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017113489999999999, 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 598 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940548+0000", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940552+0000", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.507e-06 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940563+0000", 2024-09-16T02:15:09.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1177e-05 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940571+0000", 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.763e-06 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955266+0000", 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014695142 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955563+0000", 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029711000000000001 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.957661+0000", 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002097921 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s5 72/66 ECSubWrite(tid=268, reqid=client.4598.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=72'21))", 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 8.508784554, 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51318274200000003, 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959947+0000", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959959+0000", 2024-09-16T02:15:09.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2097e-05 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471943+0000", 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51198411399999999 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471961+0000", 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8641999999999999e-05 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472084+0000", 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012220300000000001 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472100+0000", 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6583e-05 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473040+0000", 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094022500000000005 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473128+0000", 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7935000000000004e-05 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:654 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5112710419999997, 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020180719, 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 654 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957462+0000", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7800000000000001e-06 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957473+0000", 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1266e-05 2024-09-16T02:15:09.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957499+0000", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5978000000000001e-05 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957519+0000", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545999999999999e-05 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959927+0000", 2024-09-16T02:15:09.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024080640000000001 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029093199999999999 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962184+0000", 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019666520000000002 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.975499+0000", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013314994 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977423+0000", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019231490000000001 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977640+0000", 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021735799999999999 2024-09-16T02:15:09.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:655 3.8s0 3:177f2fba:::smithi19530025-45:head [rollback d] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5111843480000005, 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023374418000000001, 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 655 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957547+0000", 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957551+0000", 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1659999999999996e-06 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957562+0000", 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0773e-05 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957570+0000", 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5520000000000005e-06 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965685+0000", 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081150130000000008 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965988+0000", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030228299999999998 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968839+0000", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028510089999999998 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977674+0000", 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088351860000000001 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980727+0000", 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030533589999999998 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980920+0000", 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019316299999999999 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:656 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 6.5111479699999997, 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034162017000000003, 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 656 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.584 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957583+0000", 2024-09-16T02:15:09.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2600000000000002e-07 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957586+0000", 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1870000000000001e-06 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957595+0000", 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0189999999999995e-06 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957602+0000", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1629999999999999e-06 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972054+0000", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452242000000001 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972340+0000", 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028568300000000001 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974084+0000", 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001744034 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977813+0000", 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037288239999999999 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:09.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991646+0000", 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013832947 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991744+0000", 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8292e-05 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.589 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:09.591 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:09.590+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:09.692 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:09.690+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:09.720 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.720 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:09.720 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:09.721 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:09.792 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:09.790+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:09.858 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.858 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:09.858 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:09.858 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:09.859 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:09.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:09.893 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:09.890+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:09.992 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:09.992 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:09.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.710824154999997, 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:09.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:15:09.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:09.997 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.706601708999997, 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:09.998 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:09.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:15:10.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.706306400000003, 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006345626, 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283859+0000", 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283921+0000", 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1947999999999997e-05 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284076+0000", 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015468599999999999 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284095+0000", 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8686e-05 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285435+0000", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001340639 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285464+0000", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8411999999999998e-05 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290127+0000", 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046638529999999999 2024-09-16T02:15:10.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290204+0000", 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079000000000001e-05 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=1, reqid=client.4598.0:17, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.704272791999998, 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048188320000000003, 2024-09-16T02:15:10.007 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285892+0000", 2024-09-16T02:15:10.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285894+0000", 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.667e-06 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285942+0000", 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8133999999999997e-05 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286032+0000", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9883000000000007e-05 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286080+0000", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7605000000000002e-05 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286237+0000", 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015714200000000001 2024-09-16T02:15:10.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286262+0000", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5012e-05 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290658+0000", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043960889999999997 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290710+0000", 2024-09-16T02:15:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2299999999999997e-05 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.693849776999997, 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.691185396999998, 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:15:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.691126177000001, 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.690718871999998, 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:15:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:15:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.688838126999997, 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:15:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.688562562999998, 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:15:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:15:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.687237166000003, 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00583893, 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302929+0000", 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.717e-06 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302970+0000", 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1564999999999999e-05 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303048+0000", 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8059000000000006e-05 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303070+0000", 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282999999999999e-05 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306196+0000", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00312663 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306213+0000", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6747000000000001e-05 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308730+0000", 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002516796 2024-09-16T02:15:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308766+0000", 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6133000000000001e-05 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 38/32 ECSubWrite(tid=30, reqid=client.4598.0:199, at_version=38'21, trim_to=0'0, roll_forward_to=36'19))", 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 42.640030914999997, 2024-09-16T02:15:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046039000000000002, 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350133+0000", 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350134+0000", 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350792+0000", 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065815600000000002 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350844+0000", 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1777999999999998e-05 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.350861+0000", 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6826000000000001e-05 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351025+0000", 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164623 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.351055+0000", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9672999999999999e-05 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354677+0000", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036216830000000001 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.354737+0000", 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0534e-05 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=45, reqid=client.4598.0:244, at_version=44'36, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.570699333, 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013820059000000001, 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419467+0000", 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9400000000000001e-06 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419693+0000", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000226312 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431387+0000", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011694154999999999 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431404+0000", 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6945000000000001e-05 2024-09-16T02:15:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431560+0000", 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015586199999999999 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431575+0000", 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4979e-05 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433182+0000", 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016072700000000001 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433285+0000", 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010259600000000001 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=46, reqid=client.4598.0:245, at_version=44'38, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.559082218999997, 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060040980000000002, 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431084+0000", 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431346+0000", 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026225100000000002 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434399+0000", 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030528320000000001 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434422+0000", 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3326e-05 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434546+0000", 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411200000000001 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434562+0000", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5075999999999999e-05 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437029+0000", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024676699999999999 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437086+0000", 2024-09-16T02:15:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7008e-05 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=48, reqid=client.4598.0:247, at_version=44'40, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.555143125000001, 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29780827999999998, 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435023+0000", 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-16T02:15:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435063+0000", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731424+0000", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29636148400000001 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731439+0000", 2024-09-16T02:15:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.453e-05 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731595+0000", 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015660699999999999 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731614+0000", 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8243000000000001e-05 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732758+0000", 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011446360000000001 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732829+0000", 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1191000000000003e-05 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.633270257, 2024-09-16T02:15:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080490270000000003, 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356895+0000", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356948+0000", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3032000000000003e-05 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357075+0000", 2024-09-16T02:15:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012766899999999999 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357100+0000", 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5086e-05 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357282+0000", 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018186299999999999 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7348999999999999e-05 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364849+0000", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075397019999999997 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364943+0000", 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3709999999999996e-05 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.628518864, 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070518059999999999, 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361646+0000", 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361693+0000", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6882000000000003e-05 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361805+0000", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112046 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361834+0000", 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9215999999999999e-05 2024-09-16T02:15:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362268+0000", 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043351899999999999 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362294+0000", 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6142000000000001e-05 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368607+0000", 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063134869999999996 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9844000000000002e-05 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.626443948000002, 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059443869999999998, 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363722+0000", 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.733e-06 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363805+0000", 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2776999999999997e-05 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363906+0000", 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010156 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363931+0000", 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4966000000000001e-05 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365907+0000", 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019751790000000001 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365933+0000", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205999999999999e-05 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369569+0000", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036364890000000001 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369665+0000", 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5476999999999994e-05 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 62/61 ECSubWrite(tid=18, reqid=client.4598.0:512, at_version=62'109, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.974417895999999, 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048170599999999997, 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015746+0000", 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.015748+0000", 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.934e-06 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016438+0000", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069006799999999995 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016818+0000", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038001400000000002 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016836+0000", 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7214000000000002e-05 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016964+0000", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012812000000000001 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016981+0000", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7065999999999999e-05 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.020510+0000", 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035291950000000002 2024-09-16T02:15:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.020563+0000", 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3449000000000003e-05 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 62/61 ECSubWrite(tid=19, reqid=client.4598.0:513, at_version=62'110, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.973716727999999, 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004743438, 2024-09-16T02:15:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016448+0000", 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016449+0000", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6500000000000001e-06 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016746+0000", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029719700000000002 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016886+0000", 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013917000000000001 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.016902+0000", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6633000000000002e-05 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.018291+0000", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013883089999999999 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.018321+0000", 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0318999999999999e-05 2024-09-16T02:15:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.021129+0000", 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002807684 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:51.021191+0000", 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2476000000000003e-05 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.085 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:10.087 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:15:10.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:280: done (6 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:282: done (5 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:283: done (4 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:284: done (3 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:285: done (2 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:286: done (1 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:287: done (0 left) 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:288: rollback oid 35 current snap is 22 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 35 to 21 2024-09-16T02:15:10.088 INFO:tasks.rados.rados.0.smithi195.stdout:289: snap_create 2024-09-16T02:15:10.089 INFO:tasks.rados.rados.0.smithi195.stdout:288: finishing rollback tid 0 to smithi19530025-35 2024-09-16T02:15:10.089 INFO:tasks.rados.rados.0.smithi195.stdout:288: finishing rollback tid 1 to smithi19530025-35 2024-09-16T02:15:10.089 INFO:tasks.rados.rados.0.smithi195.stdout:288: finishing rollback tid 2 to smithi19530025-35 2024-09-16T02:15:10.089 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 35 v 110 (ObjNum 114 snap 21 seq_num 114) dirty exists 2024-09-16T02:15:10.089 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:10.082+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:10.187 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:10.185+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:10.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:10.217 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:10.217 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:10.217 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:10.217 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:10.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:10.287 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:10.282+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:10.370 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:10.370 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:10.388 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:10.385+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:10.395 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:10.397 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:10.397 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:10.397 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:10.397 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:10.397 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:10.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:10.427 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:10.427 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:10.460 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:10.461 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:10.491 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:10.498 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:10.497+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:10.523 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:10.523 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:10.577 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:10.577 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:10.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:10.577 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 64.503041920000001, 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:10.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:15:10.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:15:10.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:52 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.998271852999999, 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020204412000000001, 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.585 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 52 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576032+0000", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576033+0000", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6599999999999995e-07 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576055+0000", 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1931999999999999e-05 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576066+0000", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1369000000000001e-05 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.576074+0000", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5630000000000003e-06 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594265+0000", 2024-09-16T02:15:10.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018191161000000001 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594479+0000", 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021409700000000001 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.594685+0000", 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000205708 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.595917+0000", 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232564 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596058+0000", 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014070900000000001 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596135+0000", 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6878000000000005e-05 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.596237+0000", 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101665 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:10.591 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.776965322999999, 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019987173, 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 98 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:10.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797339+0000", 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797340+0000", 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.593 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797353+0000", 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3156000000000001e-05 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797372+0000", 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8382000000000001e-05 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797386+0000", 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4262e-05 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811114+0000", 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013728557000000001 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811354+0000", 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023977499999999999 2024-09-16T02:15:10.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.811632+0000", 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027806899999999998 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815429+0000", 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037971509999999999 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817182+0000", 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001752382 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817262+0000", 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0240000000000004e-05 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817326+0000", 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4022000000000005e-05 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.597 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.776893841000003, 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020141841000000001, 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.598 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 99 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797410+0000", 2024-09-16T02:15:10.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797412+0000", 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.243e-06 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797542+0000", 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013077400000000001 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797562+0000", 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9644e-05 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797577+0000", 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5218999999999999e-05 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812322+0000", 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014744597999999999 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812456+0000", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134466 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.812671+0000", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000214537 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815454+0000", 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027836179999999999 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817415+0000", 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019603979999999999 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817490+0000", 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5012000000000006e-05 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817552+0000", 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2331999999999999e-05 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.776701136, 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020132852, 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.604 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 100 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797603+0000", 2024-09-16T02:15:10.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797605+0000", 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7069999999999999e-06 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797620+0000", 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5585000000000001e-05 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797639+0000", 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8212000000000001e-05 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.797658+0000", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9514999999999999e-05 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.813882+0000", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016224045999999999 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814108+0000", 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022626600000000001 2024-09-16T02:15:10.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.814462+0000", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035324400000000001 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.815978+0000", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015162000000000001 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817614+0000", 2024-09-16T02:15:10.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001636366 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817684+0000", 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9701000000000002e-05 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.817736+0000", 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2009999999999998e-05 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.665186114999997, 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.611 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:15:10.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:15:10.616 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 63.665061721000001, 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:10.617 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:15:10.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:15:10.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:1 3.3s0 3:c481b30a:::smithi19530025-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.296909083999999, 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021556188, 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.623 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:10.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277399+0000", 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4400000000000001e-06 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277527+0000", 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012811100000000001 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277557+0000", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0645000000000002e-05 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277594+0000", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6848999999999999e-05 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277805+0000", 2024-09-16T02:15:10.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021101299999999999 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.278199+0000", 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039380300000000002 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.280065+0000", 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018661680000000001 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286512+0000", 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064472219999999998 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298756+0000", 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012244026 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298951+0000", 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194911 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:2 3.3s0 3:c481b30a:::smithi19530025-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.296655215000001, 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022809754000000002, 2024-09-16T02:15:10.629 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277650+0000", 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277657+0000", 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277675+0000", 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8148999999999999e-05 2024-09-16T02:15:10.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277691+0000", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5943e-05 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282023+0000", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004332247 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282308+0000", 2024-09-16T02:15:10.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028409399999999998 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285666+0000", 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033580379999999998 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.292774+0000", 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071080379999999997 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300235+0000", 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074610759999999996 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300459+0000", 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224033 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.296490542999997, 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.636 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:15:10.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:10.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.641 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.293012150000003, 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:15:10.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:15:10.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.291889976999997, 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853072, 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.648 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282416+0000", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282453+0000", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7261000000000001e-05 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298874+0000", 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016420846999999999 2024-09-16T02:15:10.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298893+0000", 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9695e-05 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299501+0000", 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000607509 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299522+0000", 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1039e-05 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303194+0000", 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036718020000000001 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303267+0000", 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3598999999999998e-05 2024-09-16T02:15:10.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.291369140999997, 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:15:10.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.660 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.660 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:15:10.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:15:10.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.291272921999997, 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:15:10.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:58 3.2s0 3:431ee5ca:::smithi19530025-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.259243568000002, 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021031579000000002, 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.667 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 58 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315062+0000", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315685+0000", 2024-09-16T02:15:10.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062327200000000002 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315711+0000", 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5519000000000001e-05 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315731+0000", 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9775e-05 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322701+0000", 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069698679999999997 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323044+0000", 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034300800000000001 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324822+0000", 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017778659999999999 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.329203+0000", 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043815149999999999 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.335988+0000", 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067852160000000002 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.336092+0000", 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103927 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:59 3.2s0 3:431ee5ca:::smithi19530025-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.258533667000002, 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022592232, 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.673 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 59 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8729999999999999e-06 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315782+0000", 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2629999999999992e-06 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315822+0000", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0095000000000002e-05 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315843+0000", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0905999999999999e-05 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326893+0000", 2024-09-16T02:15:10.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011050249999999999 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.327039+0000", 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145723 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328011+0000", 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097276899999999996 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.330713+0000", 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027020989999999999 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338142+0000", 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074284329999999999 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338363+0000", 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022082100000000001 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.678 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 46.258434500999996, 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:10.679 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:10.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:15:10.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:15:10.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.683 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:15:10.684 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.206698754000001, 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:10.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:15:10.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:15:10.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:10.689 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.202411073999997, 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:10.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.693 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 2.507856511, 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27886435399999998, 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.694 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066449+0000", 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.435e-06 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066471+0000", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1526999999999999e-05 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343774+0000", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27730348599999999 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343789+0000", 2024-09-16T02:15:10.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4946000000000001e-05 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343981+0000", 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019161199999999999 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.344000+0000", 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545e-05 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345225+0000", 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012244479999999999 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345312+0000", 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7355000000000006e-05 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:10.698 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:10.700 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:10.700 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:10.700 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:10.694+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:10.701 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:10.702 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:10.702 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:10.740 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:10.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:10.781 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:10.800 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:10.794+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:10.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:10.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:10.831 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:10.831 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:10.845 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:10.861 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:10.862 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:10.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:10.901 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:10.901 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:10.894+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:10.938 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 11] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:15:10.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:10.938 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard started 2024-09-16T02:15:10.938 INFO:tasks.thrashosds.thrasher:Killing osd 10, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-16T02:15:10.939 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T02:15:10.939 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:15:10.939 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:10.939 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:10.968 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:10.969 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:10.970+0000 7efe03fff640 0 -- 172.21.15.3:0/2507006133 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7efe0c224520 msgr2=0x7efe0c226910 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:10.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:10.985 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:11.003 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:10.998+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:11.012 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T02:15:11.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 10 2024-09-16T02:15:11.031 INFO:tasks.rados.rados.0.smithi195.stdout:288: done (1 left) 2024-09-16T02:15:11.031 INFO:tasks.rados.rados.0.smithi195.stdout:289: done (0 left) 2024-09-16T02:15:11.031 INFO:tasks.rados.rados.0.smithi195.stdout:290: copy_from oid 34 from oid 4 current snap is 23 2024-09-16T02:15:11.031 INFO:tasks.rados.rados.0.smithi195.stdout:291: rollback oid 31 current snap is 23 2024-09-16T02:15:11.031 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 31 to 21 2024-09-16T02:15:11.032 INFO:tasks.rados.rados.0.smithi195.stdout:292: copy_from oid 7 from oid 21 current snap is 23 2024-09-16T02:15:11.032 INFO:tasks.rados.rados.0.smithi195.stdout:293: copy_from oid 23 from oid 42 current snap is 23 2024-09-16T02:15:11.032 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:11.032 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:11.033 INFO:tasks.rados.rados.0.smithi195.stdout:294: delete oid 33 current snap is 23 2024-09-16T02:15:11.041 INFO:tasks.rados.rados.0.smithi195.stdout:291: finishing rollback tid 1 to smithi19530025-31 2024-09-16T02:15:11.042 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 42 (ObjNum 353 snap 0 seq_num 3649559672) dirty dne 2024-09-16T02:15:11.046 INFO:tasks.rados.rados.0.smithi195.stdout:293: finishing copy_from racing read to smithi19530025-23 2024-09-16T02:15:11.049 INFO:tasks.rados.rados.0.smithi195.stdout:293: finishing copy_from to smithi19530025-23 2024-09-16T02:15:11.049 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 23 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:11.104 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:11.098+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:11.181 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.181 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.195 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:11.204 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:11.202+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:11.305 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:11.301+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:11.361 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-09-16T02:15:11.354+0000 7f5f54a8b640 -1 mon.a@0(leader).osd e82 definitely_dead 0 2024-09-16T02:15:11.361 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10 is already down. 2024-09-16T02:15:11.363 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.363 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:11.381 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 11] 2024-09-16T02:15:11.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 10 2024-09-16T02:15:11.405 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:11.402+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:11.501 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.501 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.506 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:11.502+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:11.515 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:11.607 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:11.602+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:11.638 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.638 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.652 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:11.707 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:11.702+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:11.720 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-09-16T02:15:11.714+0000 7f5f54a8b640 -1 mon.a@0(leader).osd e82 definitely_dead 0 2024-09-16T02:15:11.770 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.771 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.785 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:11.808 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:11.802+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:11.909 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:11.909+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:11.954 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:11.954 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:11.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:12.013 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:12.010+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:12.040 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.10. 2024-09-16T02:15:12.069 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.10 2024-09-16T02:15:12.069 INFO:tasks.thrashosds.thrasher:remote for osd 10 is ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:15:12.070 DEBUG:teuthology.orchestra.run.smithi177:> type ceph-objectstore-tool 2024-09-16T02:15:12.074 INFO:teuthology.orchestra.run.smithi177.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-16T02:15:12.074 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op list-pgs 2024-09-16T02:15:12.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:15:12.096 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.096 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.110 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:12.105+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:12.111 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:12.211 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:12.206+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:12.245 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.245 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.259 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:12.312 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:12.309+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:12.413 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:12.406+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:12.426 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.426 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.441 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:12.457 INFO:teuthology.orchestra.run.smithi177.stdout:3.es3 2024-09-16T02:15:12.458 INFO:teuthology.orchestra.run.smithi177.stdout:3.bs5 2024-09-16T02:15:12.458 INFO:teuthology.orchestra.run.smithi177.stdout:3.as1 2024-09-16T02:15:12.458 INFO:teuthology.orchestra.run.smithi177.stdout:3.5s0 2024-09-16T02:15:12.458 INFO:teuthology.orchestra.run.smithi177.stdout:3.3s6 2024-09-16T02:15:12.458 INFO:teuthology.orchestra.run.smithi177.stdout:3.2s3 2024-09-16T02:15:12.514 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:12.510+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:12.558 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.558 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.572 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:12.616 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:12.614+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:12.716 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:12.710+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:12.743 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.743 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.757 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:12.759 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --op export --pgid 3.3s6 --file /var/log/ceph/exp.3.3s6.10 2024-09-16T02:15:12.817 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:12.814+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:12.877 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:12.877 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:12.891 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:12.936 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:12.934+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:13.018 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:13.014+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:13.050 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:13.057 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:13.057 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:13.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:13.077 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:291: done (4 left) 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:293: done (3 left) 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:294: done (2 left) 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:295: rollback oid 29 current snap is 23 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 29 to 22 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 20 current snap is 23 2024-09-16T02:15:13.090 INFO:tasks.rados.rados.0.smithi195.stdout:296: seq_num 119 ranges {2113536=458752,2572288=507904,3080192=507904,3588096=458752,4046848=81920} 2024-09-16T02:15:13.108 INFO:tasks.rados.rados.0.smithi195.stdout:296: writing smithi19530025-20 from 2113536 to 2572288 tid 1 2024-09-16T02:15:13.112 INFO:tasks.rados.rados.0.smithi195.stdout:296: writing smithi19530025-20 from 2572288 to 3080192 tid 2 2024-09-16T02:15:13.116 INFO:tasks.rados.rados.0.smithi195.stdout:296: writing smithi19530025-20 from 3080192 to 3588096 tid 3 2024-09-16T02:15:13.119 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:13.114+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:13.119 INFO:tasks.rados.rados.0.smithi195.stdout:296: writing smithi19530025-20 from 3588096 to 4046848 tid 4 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:296: writing smithi19530025-20 from 4046848 to 4128768 tid 5 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:297: rollback oid 6 current snap is 23 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 6 to 11 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:290: finishing copy_from to smithi19530025-34 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:290: got expected ENOENT (src dne) 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:290: finishing copy_from racing read to smithi19530025-34 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:292: finishing copy_from racing read to smithi19530025-7 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:295: finishing rollback tid 0 to smithi19530025-29 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:295: finishing rollback tid 1 to smithi19530025-29 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:295: finishing rollback tid 2 to smithi19530025-29 2024-09-16T02:15:13.121 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 29 v 48 (ObjNum 117 snap 21 seq_num 117) dirty exists 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:290: done (4 left) 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:295: done (3 left) 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:298: read oid 15 snap -1 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:298: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:299: read oid 31 snap -1 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:299: expect deleted 2024-09-16T02:15:13.122 INFO:tasks.rados.rados.0.smithi195.stdout:300: snap_remove snap 22 2024-09-16T02:15:13.148 INFO:teuthology.orchestra.run.smithi177.stderr:Exporting 3.3s6 info 3.3s6( v 72'21 (0'0,72'21] local-lis/les=66/67 n=2 ec=32/32 lis/c=66/66 les/c/f=67/67/0 sis=66) 2024-09-16T02:15:13.149 INFO:teuthology.orchestra.run.smithi177.stderr:Read 6#3:c481b30a:::smithi19530025-1:head# 2024-09-16T02:15:13.150 INFO:teuthology.orchestra.run.smithi177.stderr:Read 6#3:c628ba0c:::smithi19530025-27:head# 2024-09-16T02:15:13.150 INFO:teuthology.orchestra.run.smithi177.stderr:Export successful 2024-09-16T02:15:13.207 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:13.208 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:13.220 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:13.217+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:13.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:13.246 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:13.247 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:13.272 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:13.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:13.301 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:13.321 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:13.317+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:13.329 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:13.329 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:13.357 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:13.357 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:13.387 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:13.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:13.389 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:13.389 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:13.404 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:13.418 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:13.419 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:13.441 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:13.441 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:13.452 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:13.450+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:13.456 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal --force --op remove --pgid 3.3s6 2024-09-16T02:15:13.470 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:13.471 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:13.502 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:13.502 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:13.503 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:13.521 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:13.517+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:13.529 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:13.529 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:13.536 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:13.537 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:13.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:13.568 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:13.568 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:13.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:13.603 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:13.632 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:13.630+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:13.637 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:13.670 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:13.670 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:13.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:13.708 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:13.708 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:13.709 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:13.709 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:13.723 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:13.723 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:13.718+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:13.824 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:13.821+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:13.887 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:13.887 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:13.887 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:13.888 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:13.888 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:13.902 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:13.924 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:13.918+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:14.025 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:14.022+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:14.028 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:14.028 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:14.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:14.028 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:14.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.745840094999998, 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.029 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.745647320000003, 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:15:14.035 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:14.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:15:14.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:15:14.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.745589277000001, 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:15:14.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:15:14.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:14.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:15:14.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:15:14.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.048 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.716432347000001, 2024-09-16T02:15:14.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.050 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:14.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:15:14.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:15:14.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:15:14.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.716379287999999, 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.059 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:15:14.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.681197824999998, 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.064 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:15:14.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 38.667985127000001, 2024-09-16T02:15:14.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:15:14.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:15:14.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 38.667875889999998, 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.076 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:14.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:15:14.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:14.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:15:14.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:14.082 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.118611709, 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:15:14.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:15:14.087 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.118495981999999, 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.088 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:15:14.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:15:14.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:15:14.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.096961198999999, 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:15:14.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:15:14.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:537 3.6s0 3:63499ffe:::smithi19530025-18:head [read 18069~227691,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 22.027927385000002, 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012782834999999999, 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 537 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.097 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997148+0000", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997149+0000", 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.203e-06 2024-09-16T02:15:14.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997156+0000", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.2130000000000002e-06 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997174+0000", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7422999999999999e-05 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997188+0000", 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.45e-05 2024-09-16T02:15:14.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997411+0000", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022312100000000001 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:51.997421+0000", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9469999999999998e-06 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.006991+0000", 2024-09-16T02:15:14.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0095694119999999994 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.007127+0000", 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013637400000000001 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:52.009931+0000", 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028036419999999999 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 73/66 ECSubWrite(tid=66, reqid=osd.8.0:66, at_version=73'42, trim_to=0'0, roll_forward_to=72'38))", 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.070129154, 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018190284000000001, 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954947+0000", 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5069999999999999e-06 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954992+0000", 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4162000000000002e-05 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955233+0000", 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000241566 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955251+0000", 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7448999999999998e-05 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956483+0000", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012321509999999999 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956507+0000", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973035+0000", 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016528547000000001 2024-09-16T02:15:14.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973136+0000", 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010087200000000001 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 73/32 ECSubWrite(tid=107, reqid=osd.5.0:107, at_version=73'75, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.066420613, 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016895095999999998, 2024-09-16T02:15:14.106 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958656+0000", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5939999999999999e-06 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958707+0000", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1008999999999999e-05 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958930+0000", 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022282899999999999 2024-09-16T02:15:14.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958952+0000", 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2146000000000001e-05 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959126+0000", 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173807 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959156+0000", 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0022000000000002e-05 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975497+0000", 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016340924999999999 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975550+0000", 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2763999999999998e-05 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.110 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs6 73/32 ECSubWrite(tid=80, reqid=osd.14.0:80, at_version=73'29, trim_to=0'0, roll_forward_to=70'27))", 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.963198+0000", 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.061877267, 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014260551999999999, 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963198+0000", 2024-09-16T02:15:14.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963198+0000", 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963198+0000", 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3200000000000001e-07 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963247+0000", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.871e-05 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963307+0000", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0092e-05 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963318+0000", 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0818999999999999e-05 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963408+0000", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9846999999999995e-05 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963419+0000", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1377e-05 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.977420+0000", 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014001112 2024-09-16T02:15:14.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.977458+0000", 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8362999999999998e-05 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.060974358999999, 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014562088000000001, 2024-09-16T02:15:14.115 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964103+0000", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.26e-06 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964341+0000", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023783300000000001 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964452+0000", 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110974 2024-09-16T02:15:14.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964476+0000", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.393e-05 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964624+0000", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014846400000000001 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964652+0000", 2024-09-16T02:15:14.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7389000000000002e-05 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978619+0000", 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013967228 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978663+0000", 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.401e-05 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 73/32 ECSubWrite(tid=76, reqid=osd.10.0:76, at_version=73'44, trim_to=0'0, roll_forward_to=71'42))", 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964852+0000", 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.060223276, 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014002486999999999, 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964852+0000", 2024-09-16T02:15:14.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964852+0000", 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964852+0000", 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1699999999999998e-07 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964903+0000", 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0739000000000003e-05 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965011+0000", 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000108328 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965033+0000", 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1929e-05 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965188+0000", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015435399999999999 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965245+0000", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7169000000000001e-05 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978796+0000", 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01355135 2024-09-16T02:15:14.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978854+0000", 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.8100999999999998e-05 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=104, reqid=client.4598.0:647, at_version=73'29, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.968800+0000", 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.056275313, 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011194668, 2024-09-16T02:15:14.124 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968800+0000", 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968800+0000", 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968801+0000", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4440000000000001e-06 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968898+0000", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6698000000000002e-05 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969000+0000", 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010164 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969014+0000", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.484e-05 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.976468+0000", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0074532419999999997 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.976480+0000", 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.235e-05 2024-09-16T02:15:14.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.979955+0000", 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003475349 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.979994+0000", 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9104999999999999e-05 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.128 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=107, reqid=client.4598.0:648, at_version=73'30, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.969510+0000", 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.05556494, 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010511306999999999, 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969510+0000", 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969510+0000", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969511+0000", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7800000000000002e-07 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969806+0000", 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000295378 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969862+0000", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6310999999999997e-05 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.969876+0000", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3998999999999999e-05 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.976955+0000", 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0070787259999999996 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.976966+0000", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1141999999999999e-05 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.979998+0000", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003031761 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.980021+0000", 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3512000000000001e-05 2024-09-16T02:15:14.132 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.9592443289999997, 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79668697799999999, 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.133 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065832+0000", 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4780000000000001e-06 2024-09-16T02:15:14.134 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065862+0000", 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9570999999999999e-05 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860567+0000", 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79470507099999999 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.135 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860588+0000", 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0665e-05 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860756+0000", 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016797700000000001 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860787+0000", 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1492000000000003e-05 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.136 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862431+0000", 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016435880000000001 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862518+0000", 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7136000000000001e-05 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:14.137 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:14.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:300: done (5 left) 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:301: read oid 36 snap -1 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:301: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:302: setattr oid 34 current snap is 23 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:292: finishing copy_from to smithi19530025-7 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 52 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:297: finishing rollback tid 1 to smithi19530025-6 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:298: expect (ObjNum 14 snap 0 seq_num 14) 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 194126 2024-09-16T02:15:14.142 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 42842 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:297: finishing rollback tid 2 to smithi19530025-6 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 6 v 25 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 1 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 2 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 3 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 4 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 5 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 6 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: finishing write tid 7 to smithi19530025-20 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 updating version 0 to 104 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 version 104 is already newer than 102 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 version 104 is already newer than 101 2024-09-16T02:15:14.143 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 updating version 104 to 105 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 updating version 105 to 106 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:296: oid 20 version 106 is already newer than 103 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 20 v 106 (ObjNum 119 snap 23 seq_num 119) dirty exists 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:296: left oid 20 (ObjNum 119 snap 23 seq_num 119) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:292: done (6 left) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:296: done (5 left) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:297: done (4 left) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:298: done (3 left) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:299: done (2 left) 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:303: rollback oid 41 current snap is 23 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 41 to 11 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 31 current snap is 23 2024-09-16T02:15:14.144 INFO:tasks.rados.rados.0.smithi195.stdout:304: seq_num 120 ranges {0=475136,475136=655360,1130496=622592,1753088=278528} 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:304: writing smithi19530025-31 from 0 to 475136 tid 1 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:304: writing smithi19530025-31 from 475136 to 1130496 tid 2 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:304: writing smithi19530025-31 from 1130496 to 1753088 tid 3 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:304: writing smithi19530025-31 from 1753088 to 2031616 tid 4 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:305: snap_create 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:303: finishing rollback tid 0 to smithi19530025-41 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:301: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 80856 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:303: finishing rollback tid 1 to smithi19530025-41 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:303: finishing rollback tid 2 to smithi19530025-41 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 41 v 152 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-16T02:15:14.145 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 1 to smithi19530025-31 2024-09-16T02:15:14.146 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 2 to smithi19530025-31 2024-09-16T02:15:14.146 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 3 to smithi19530025-31 2024-09-16T02:15:14.146 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 4 to smithi19530025-31 2024-09-16T02:15:14.146 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 108 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-16T02:15:14.146 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:14.134+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:14.240 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:14.238+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:14.279 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:14.280 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:14.280 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:14.280 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:14.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:14.341 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:14.337+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:14.424 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:14.424 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:14.424 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:14.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:14.441 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:14.438+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:14.542 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:14.538+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:14.564 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:76 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'54, mlcod=24'46)", 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.874209018000002, 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007242999, 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687520+0000", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.796e-06 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687531+0000", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1420000000000001e-05 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687607+0000", 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5166999999999994e-05 2024-09-16T02:15:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687620+0000", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3873000000000001e-05 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689759+0000", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021384210000000002 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689788+0000", 2024-09-16T02:15:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8782999999999999e-05 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694680+0000", 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004892123 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694761+0000", 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1415999999999994e-05 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:96 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'70, mlcod=24'65)", 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.755359697000003, 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063466429999999999, 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806368+0000", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806369+0000", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.297e-06 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806477+0000", 2024-09-16T02:15:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107983 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806532+0000", 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5075999999999999e-05 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806544+0000", 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2405999999999999e-05 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806623+0000", 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8689999999999994e-05 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.806656+0000", 2024-09-16T02:15:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3102999999999998e-05 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812648+0000", 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059919650000000001 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.812714+0000", 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6123000000000004e-05 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'87)", 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.639412594999996, 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069325319999999999, 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922316+0000", 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4640000000000001e-06 2024-09-16T02:15:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922381+0000", 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4969999999999996e-05 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924466+0000", 2024-09-16T02:15:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002085075 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924479+0000", 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3033e-05 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924623+0000", 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924687+0000", 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3737000000000003e-05 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929168+0000", 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044810020000000004 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929247+0000", 2024-09-16T02:15:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9472000000000001e-05 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'89)", 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.637306885000001, 2024-09-16T02:15:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33740482100000002, 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924422+0000", 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924431+0000", 2024-09-16T02:15:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8769999999999999e-06 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260013+0000", 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33558222100000001 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260031+0000", 2024-09-16T02:15:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8063e-05 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260228+0000", 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000196782 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260273+0000", 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877000000000001e-05 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261745+0000", 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001471916 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261825+0000", 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9791999999999998e-05 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.204722142000001, 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071696049999999999, 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357006+0000", 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9399999999999995e-07 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357033+0000", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6978000000000002e-05 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357063+0000", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0549999999999997e-05 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357074+0000", 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1077000000000001e-05 2024-09-16T02:15:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357740+0000", 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066576700000000005 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357753+0000", 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2706e-05 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364120+0000", 2024-09-16T02:15:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063668939999999997 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364175+0000", 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5139000000000002e-05 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.201985223999998, 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068742350000000002, 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0699999999999998e-07 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359769+0000", 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6655000000000001e-05 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359836+0000", 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7043000000000002e-05 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359848+0000", 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.199e-05 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359928+0000", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9927999999999998e-05 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359940+0000", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.202e-05 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366543+0000", 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066029269999999998 2024-09-16T02:15:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366616+0000", 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3264999999999997e-05 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.201818369999998, 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067033939999999997, 2024-09-16T02:15:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359909+0000", 2024-09-16T02:15:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359910+0000", 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.22e-06 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359939+0000", 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9213999999999998e-05 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359987+0000", 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7602999999999998e-05 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359999+0000", 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1666e-05 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360077+0000", 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8336000000000004e-05 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360089+0000", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2432999999999999e-05 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366574+0000", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064847710000000003 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366612+0000", 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8151000000000001e-05 2024-09-16T02:15:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.199648897000003, 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080847929999999998, 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362079+0000", 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.054e-06 2024-09-16T02:15:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362106+0000", 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6322e-05 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362152+0000", 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6276999999999998e-05 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362163+0000", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.064e-05 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362225+0000", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2539999999999994e-05 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362236+0000", 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1151999999999999e-05 2024-09-16T02:15:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370090+0000", 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00785368 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370163+0000", 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3127999999999995e-05 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 66/32 ECSubWrite(tid=73, reqid=client.4598.0:580, at_version=66'64, trim_to=0'0, roll_forward_to=66'59))", 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 19.512212093999999, 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51414269899999998, 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049516+0000", 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.68e-07 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049529+0000", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2897e-05 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562582+0000", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51305329300000002 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562593+0000", 2024-09-16T02:15:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1092e-05 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562676+0000", 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2935000000000005e-05 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562686+0000", 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0067e-05 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563613+0000", 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092661900000000005 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563658+0000", 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5127999999999999e-05 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=271, reqid=osd.13.0:271, at_version=73'80, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.610151993000001, 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071430060000000004, 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951576+0000", 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4700000000000001e-07 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951631+0000", 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5352999999999997e-05 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951737+0000", 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106183 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951760+0000", 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2526000000000001e-05 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951944+0000", 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000184112 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951971+0000", 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6573000000000001e-05 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958634+0000", 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066636960000000002 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958718+0000", 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3916000000000001e-05 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.608957081, 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073114410000000001, 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952772+0000", 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3300000000000001e-06 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952824+0000", 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1836000000000003e-05 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952926+0000", 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101223 2024-09-16T02:15:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952948+0000", 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2478999999999999e-05 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953117+0000", 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016898299999999999 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953144+0000", 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7036e-05 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.959979+0000", 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068347410000000001 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.960082+0000", 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010281300000000001 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=274, reqid=osd.13.0:274, at_version=73'84, trim_to=0'0, roll_forward_to=73'82))", 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.247692489, 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047083613000000003, 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314037+0000", 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7940000000000001e-06 2024-09-16T02:15:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314080+0000", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3053000000000003e-05 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359609+0000", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045529636999999998 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359632+0000", 2024-09-16T02:15:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2869000000000001e-05 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359778+0000", 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014631299999999999 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359798+0000", 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9099000000000001e-05 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361025+0000", 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012279420000000001 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361118+0000", 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2905999999999994e-05 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.954966+0000", 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.606760747999999, 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065434330000000004, 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.954966+0000", 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.954966+0000", 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.954967+0000", 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-07 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955011+0000", 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4323000000000002e-05 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955165+0000", 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015346899999999999 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955189+0000", 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4513000000000001e-05 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955355+0000", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000165709 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955376+0000", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1549999999999999e-05 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961424+0000", 2024-09-16T02:15:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060474429999999996 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961510+0000", 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6002999999999996e-05 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.602889873000001, 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010173534, 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958839+0000", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7570000000000001e-06 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958892+0000", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2469999999999997e-05 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958988+0000", 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6779000000000003e-05 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.959011+0000", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.245e-05 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961665+0000", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026538410000000001 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961690+0000", 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5313e-05 2024-09-16T02:15:14.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968914+0000", 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072245180000000001 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6405999999999998e-05 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.640 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.600266048, 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011431721000000001, 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:14.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961463+0000", 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.68e-06 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961515+0000", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2586000000000001e-05 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961613+0000", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7598999999999998e-05 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961635+0000", 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1780000000000002e-05 2024-09-16T02:15:14.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962881+0000", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012456850000000001 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962897+0000", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6303e-05 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972785+0000", 2024-09-16T02:15:14.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098886259999999993 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972893+0000", 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107462 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.2s6 84/83 ECSubRead(tid=314, to_read={3:476b4430:::smithi19530025-30:14=262144,262144,0}, subchunks={3:476b4430:::smithi19530025-30:14=[0,1]}, attrs_to_read=))", 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.332655656, 2024-09-16T02:15:14.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57667217800000004, 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229083+0000", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0162e-05 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.804968+0000", 2024-09-16T02:15:14.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57588503999999996 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805000+0000", 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2283000000000002e-05 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805176+0000", 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017626499999999999 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805744+0000", 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056751299999999996 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=159, reqid=client.4598.0:757, at_version=85'150, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082513+0000", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.47921414200000001, 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006398155, 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082513+0000", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082513+0000", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082514+0000", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5400000000000001e-07 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082575+0000", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1356e-05 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082777+0000", 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020220500000000001 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082804+0000", 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6809999999999999e-05 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083005+0000", 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020106200000000001 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083040+0000", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.481e-05 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.088828+0000", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057883539999999999 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.088911+0000", 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2904000000000004e-05 2024-09-16T02:15:14.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=160, reqid=client.4598.0:758, at_version=85'151, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.47914566600000003, 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092037909999999994, 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000001e-07 2024-09-16T02:15:14.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082613+0000", 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0790000000000002e-05 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082884+0000", 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027123099999999998 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082902+0000", 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.753e-05 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085344+0000", 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024427120000000001 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085371+0000", 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6587000000000001e-05 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091675+0000", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063039510000000003 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091785+0000", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110436 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=161, reqid=client.4598.0:759, at_version=85'152, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:14.658 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.47911016200000001, 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010507528, 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082618+0000", 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082651+0000", 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3246e-05 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083064+0000", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041363000000000001 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083085+0000", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.048e-05 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089109+0000", 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060238330000000001 2024-09-16T02:15:14.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089137+0000", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8569999999999999e-05 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093029+0000", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038919520000000002 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093125+0000", 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5307000000000001e-05 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.662 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 85/83 ECSubWrite(tid=320, reqid=client.4598.0:756, at_version=85'108, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.469438998, 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076015120000000004, 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:14.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:14.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092289+0000", 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9599999999999998e-07 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092436+0000", 2024-09-16T02:15:14.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000146983 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096144+0000", 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037081829999999999 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096168+0000", 2024-09-16T02:15:14.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3782999999999999e-05 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096349+0000", 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018129600000000001 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096382+0000", 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2252000000000002e-05 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099809+0000", 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034276129999999999 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099890+0000", 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0605999999999994e-05 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:14.669 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:14.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:14.671 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:14.666+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:14.690 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 5 to smithi19530025-31 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: finishing write tid 6 to smithi19530025-31 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: oid 31 updating version 0 to 59 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: oid 31 updating version 59 to 61 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: oid 31 version 61 is already newer than 57 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: oid 31 version 61 is already newer than 58 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: oid 31 version 61 is already newer than 60 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 61 (ObjNum 120 snap 23 seq_num 120) dirty exists 2024-09-16T02:15:14.693 INFO:tasks.rados.rados.0.smithi195.stdout:304: left oid 31 (ObjNum 120 snap 23 seq_num 120) 2024-09-16T02:15:14.771 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:14.766+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:14.842 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:14.842 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:14.843 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:14.843 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:14.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:14.872 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:14.869+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:14.973 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:14.970+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:15.027 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.027 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:15.027 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:15.027 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:15.027 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:15.053 INFO:teuthology.orchestra.run.smithi177.stdout: marking collection for removal 2024-09-16T02:15:15.053 INFO:teuthology.orchestra.run.smithi177.stdout:setting '_remove' omap key 2024-09-16T02:15:15.053 INFO:teuthology.orchestra.run.smithi177.stdout:finish_remove_pgs 3.3s6_head removing 3.3s6 2024-09-16T02:15:15.057 INFO:teuthology.orchestra.run.smithi177.stdout:Remove successful 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:301: done (4 left) 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:302: done (3 left) 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:303: done (2 left) 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:304: done (1 left) 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:305: done (0 left) 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:306: copy_from oid 39 from oid 42 current snap is 24 2024-09-16T02:15:15.074 INFO:tasks.rados.rados.0.smithi195.stdout:307: snap_create 2024-09-16T02:15:15.075 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:15.069+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:15.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:15.095 INFO:tasks.rados.rados.0.smithi195.stdout:306: finishing copy_from racing read to smithi19530025-39 2024-09-16T02:15:15.100 INFO:tasks.rados.rados.0.smithi195.stdout:306: finishing copy_from to smithi19530025-39 2024-09-16T02:15:15.100 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 39 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:15.175 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:15.170+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.829098350000002, 2024-09-16T02:15:15.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003304399, 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346334+0000", 2024-09-16T02:15:15.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346336+0000", 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8929999999999999e-06 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346359+0000", 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2534000000000001e-05 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346435+0000", 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6236000000000007e-05 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346451+0000", 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5707000000000001e-05 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346735+0000", 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028424199999999998 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346747+0000", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1725999999999999e-05 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349591+0000", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002844329 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349639+0000", 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7732000000000003e-05 2024-09-16T02:15:15.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.827589181, 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039104070000000003, 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347844+0000", 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9999999999999999e-07 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347874+0000", 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0167000000000001e-05 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347952+0000", 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8427e-05 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347965+0000", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2937e-05 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348056+0000", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0390999999999996e-05 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348066+0000", 2024-09-16T02:15:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0584000000000001e-05 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351696+0000", 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003630084 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351754+0000", 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7516999999999998e-05 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.825810871999998, 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046999930000000004, 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8400000000000001e-07 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349656+0000", 2024-09-16T02:15:15.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4477999999999998e-05 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349726+0000", 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9680999999999999e-05 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349739+0000", 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.323e-05 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349814+0000", 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349826+0000", 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1955e-05 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354271+0000", 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044454020000000002 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354322+0000", 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0506000000000001e-05 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:15.191 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.818201268000003, 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052360729999999999, 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357232+0000", 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357254+0000", 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2855999999999999e-05 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5995000000000002e-05 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357322+0000", 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1798e-05 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357409+0000", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7282999999999998e-05 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357425+0000", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.52e-05 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362424+0000", 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049990690000000001 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362467+0000", 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3674000000000002e-05 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.818069125000001, 2024-09-16T02:15:15.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065655959999999999, 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357364+0000", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8199999999999999e-07 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357381+0000", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7022999999999999e-05 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357419+0000", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8779999999999998e-05 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357427+0000", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357549+0000", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121402 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357561+0000", 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2807e-05 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363880+0000", 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063186550000000003 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363929+0000", 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9030999999999999e-05 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.815821974999999, 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045504229999999996, 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:15.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359612+0000", 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.122e-06 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359626+0000", 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4422e-05 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359678+0000", 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1765999999999999e-05 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359690+0000", 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2143e-05 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359766+0000", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5883000000000005e-05 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359776+0000", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0448999999999999e-05 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364111+0000", 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043348809999999996 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364161+0000", 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9756999999999998e-05 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 40.815527076999999, 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065736739999999998, 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359931+0000", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4975999999999999e-05 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359972+0000", 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0244000000000003e-05 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359982+0000", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0710000000000001e-05 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360066+0000", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3308999999999998e-05 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360080+0000", 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4363e-05 2024-09-16T02:15:15.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366432+0000", 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00635214 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366479+0000", 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6959999999999998e-05 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=38, reqid=client.4598.0:394, at_version=52'23, trim_to=0'0, roll_forward_to=48'21))", 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.267580877, 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043643559999999998, 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907890+0000", 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7839999999999997e-05 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907996+0000", 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106345 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908022+0000", 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5293999999999999e-05 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908228+0000", 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206372 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908273+0000", 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5225999999999997e-05 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912158+0000", 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038845490000000002 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912216+0000", 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8359999999999997e-05 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=54, reqid=client.4598.0:504, at_version=58'35, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.163246136000001, 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048877479999999999, 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012189+0000", 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0559999999999999e-06 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013086+0000", 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089770099999999999 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013357+0000", 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027122399999999998 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013374+0000", 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6160000000000001e-05 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013559+0000", 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185515 2024-09-16T02:15:15.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013585+0000", 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6037000000000002e-05 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016983+0000", 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033975289999999998 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017074+0000", 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1526000000000004e-05 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.220 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=55, reqid=client.4598.0:505, at_version=58'36, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.162337846, 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042762490000000002, 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1399999999999997e-07 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013218+0000", 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012275200000000001 2024-09-16T02:15:15.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013417+0000", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019934400000000001 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013427+0000", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3370000000000006e-06 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015391+0000", 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019645779999999998 2024-09-16T02:15:15.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015399+0000", 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017304+0000", 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001904953 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017371+0000", 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6846000000000003e-05 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=56, reqid=client.4598.0:506, at_version=58'37, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.162181817, 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056867990000000002, 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.225 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8999999999999997e-07 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013335+0000", 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4120000000000001e-05 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013545+0000", 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020994500000000001 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013558+0000", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3044000000000001e-05 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015997+0000", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024385710000000001 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016025+0000", 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7932999999999998e-05 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018839+0000", 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028140270000000002 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018938+0000", 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8669e-05 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=57, reqid=client.4598.0:507, at_version=58'38, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.161824516999999, 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059371409999999999, 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013609+0000", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7999999999999995e-07 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013650+0000", 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1192000000000001e-05 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013688+0000", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7902999999999999e-05 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013698+0000", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9690000000000006e-06 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017129+0000", 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034309319999999998 2024-09-16T02:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017136+0000", 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.396e-06 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019482+0000", 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002345813 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019545+0000", 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3355999999999994e-05 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 69/32 ECSubWrite(tid=83, reqid=client.4598.0:594, at_version=69'55, trim_to=0'0, roll_forward_to=67'53))", 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:57.934499+0000", 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.24093392, 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031892090000000001, 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.234 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.934499+0000", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.934499+0000", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.934499+0000", 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2300000000000002e-07 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.934949+0000", 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044963700000000002 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.935003+0000", 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4536000000000001e-05 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.935019+0000", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5582000000000001e-05 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.935133+0000", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011436 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.935149+0000", 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5858999999999998e-05 2024-09-16T02:15:15.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.937609+0000", 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024599090000000001 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:57.937688+0000", 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8903e-05 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.238 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 73/32 ECSubWrite(tid=104, reqid=osd.5.0:104, at_version=73'71, trim_to=0'0, roll_forward_to=72'69))", 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 13.232494599000001, 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051077429999999997, 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3700000000000001e-07 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943033+0000", 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4955000000000001e-05 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943150+0000", 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116166 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943181+0000", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1717000000000001e-05 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943369+0000", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018737199999999999 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943404+0000", 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5102999999999999e-05 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.947951+0000", 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045476809999999996 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948046+0000", 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4312000000000001e-05 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.243 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 73/32 ECSubWrite(tid=105, reqid=osd.5.0:105, at_version=73'73, trim_to=0'0, roll_forward_to=72'69))", 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 13.231322286999999, 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078423599999999996, 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944112+0000", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7400000000000001e-06 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944179+0000", 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6805999999999997e-05 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944269+0000", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0508999999999997e-05 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944293+0000", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.393e-05 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.945156+0000", 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086232699999999997 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.945183+0000", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7327000000000001e-05 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951860+0000", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066769079999999996 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951953+0000", 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2812999999999994e-05 2024-09-16T02:15:15.247 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.220247032, 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038733359999999998, 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.37e-07 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955246+0000", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0217000000000003e-05 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955369+0000", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012299199999999999 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955393+0000", 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3589999999999999e-05 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955564+0000", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170858 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955596+0000", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2539999999999997e-05 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958958+0000", 2024-09-16T02:15:15.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033615889999999999 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.959059+0000", 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100913 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:15.252 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.217765779, 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087020870000000007, 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:15.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957669+0000", 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.72e-06 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957733+0000", 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4912999999999993e-05 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957814+0000", 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0241999999999995e-05 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957829+0000", 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5007e-05 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957982+0000", 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153215 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958008+0000", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5927000000000001e-05 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.966293+0000", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082847939999999998 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.966369+0000", 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6268999999999998e-05 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.256 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.213832874, 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075408059999999997, 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961602+0000", 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.863e-06 2024-09-16T02:15:15.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961656+0000", 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4599999999999999e-05 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961776+0000", 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119926 2024-09-16T02:15:15.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961802+0000", 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5709999999999999e-05 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961956+0000", 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015436200000000001 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961983+0000", 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6460000000000001e-05 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969059+0000", 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070767859999999998 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969141+0000", 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1099000000000004e-05 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=113, reqid=client.4598.0:656, at_version=74'83, trim_to=0'0, roll_forward_to=74'77))", 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.202098568, 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035945349999999998, 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973336+0000", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973380+0000", 2024-09-16T02:15:15.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4026000000000002e-05 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973485+0000", 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104776 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973510+0000", 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5333e-05 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973675+0000", 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016454600000000001 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973701+0000", 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6676999999999999e-05 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.976856+0000", 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003154858 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.976929+0000", 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2318999999999997e-05 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s3 85/83 ECSubWrite(tid=320, reqid=client.4598.0:756, at_version=85'108, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.0846920120000001, 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005788835, 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.266 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090743+0000", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2199999999999999e-06 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091068+0000", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000325414 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091179+0000", 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110759 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091204+0000", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5227000000000001e-05 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091508+0000", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030419099999999999 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091551+0000", 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2215999999999997e-05 2024-09-16T02:15:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096441+0000", 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048900719999999996 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096529+0000", 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8735999999999999e-05 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.270 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.271 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.271 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:15.275 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:15.270+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:15.376 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:15.370+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:15.393 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:15.394 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:15.394 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:15.477 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:15.470+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:15.531 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:15.531 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:15.532 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:15.532 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:15.586 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:15.586+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:15.601 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-10 --journal-path /var/lib/ceph/osd/ceph-10/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3s6.10 2024-09-16T02:15:15.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:15.679 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:15.674+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 45/32 ECSubWrite(tid=34, reqid=osd.3.0:34, at_version=45'32, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:15:15.717 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 41.352765441999999, 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070140769999999996, 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362580+0000", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362582+0000", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-06 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362610+0000", 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7945000000000001e-05 2024-09-16T02:15:15.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362658+0000", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8439000000000002e-05 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362670+0000", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1960000000000001e-05 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362748+0000", 2024-09-16T02:15:15.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8162000000000001e-05 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362759+0000", 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0978999999999999e-05 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369332+0000", 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065723960000000003 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.369595+0000", 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026293399999999999 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:488 3.es0 3:701f1021:::smithi19530025-33:head [read 395661~145011,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.782503905999999, 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080293050000000005, 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.722 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 488 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932842+0000", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932845+0000", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1089999999999998e-06 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932860+0000", 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4576000000000001e-05 2024-09-16T02:15:15.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932891+0000", 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1504000000000002e-05 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932910+0000", 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8334000000000001e-05 2024-09-16T02:15:15.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933050+0000", 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014058899999999999 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933068+0000", 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7396000000000001e-05 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938059+0000", 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049910120000000004 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938174+0000", 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115674 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.940871+0000", 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026971109999999999 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:489 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.782400850999998, 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092923889999999999, 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 489 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932946+0000", 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932970+0000", 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932983+0000", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.27e-05 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932991+0000", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933148+0000", 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156828 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933157+0000", 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8409999999999995e-06 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938656+0000", 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005498541 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938725+0000", 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9061999999999997e-05 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.942237+0000", 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035124739999999998 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:490 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:15.733 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 29.782316941000001, 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010928416999999999, 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 490 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:15.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933030+0000", 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6700000000000003e-07 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933034+0000", 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7169999999999999e-06 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933046+0000", 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1584e-05 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933060+0000", 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3953e-05 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933191+0000", 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131347 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933199+0000", 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7030000000000006e-06 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939244+0000", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060455639999999998 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939311+0000", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6313e-05 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.943957+0000", 2024-09-16T02:15:15.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046465689999999997 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:492 3.es0 3:701f1021:::smithi19530025-33:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.765548290000002, 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017209518, 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:15.739 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 492 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949801+0000", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2100000000000002e-06 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949954+0000", 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015366300000000001 2024-09-16T02:15:15.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949991+0000", 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6463e-05 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950016+0000", 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4593e-05 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950156+0000", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000140882 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950172+0000", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5590000000000002e-05 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954336+0000", 2024-09-16T02:15:15.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041644309999999997 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954560+0000", 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022320200000000001 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.956344+0000", 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017841000000000001 2024-09-16T02:15:15.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.958842+0000", 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024979220000000001 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.966762+0000", 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079201440000000005 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.967007+0000", 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024531800000000001 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:653 3.es0 3:701f1021:::smithi19530025-33:head [rmxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ in=40b,rmxattr OOq^bpU^bm in=10b,rmxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX in=40b,rmxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ in=40b,rmxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU in=40b,rmxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe in=40b,rmxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn in=40b,rmxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr in=40b,rmxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ in=40b,rmxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a in=40b,rmxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp in=40b,rmxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV in=40b,rmxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS in=40b,rmxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR in=40b,rmxattr biYdnofqNd_eZg\\amb``Na_saZ^ in=27b,rmxattr fceY\\cr[bnaj[ in=13b,rmxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq in=40b,rmxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t in=40b,rmxattr olt]ab in=6b,rmxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf in=40b] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.758957269, 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025463508999999999, 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 653 2024-09-16T02:15:15.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956393+0000", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.454e-06 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956416+0000", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2685999999999999e-05 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956463+0000", 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7349999999999999e-05 2024-09-16T02:15:15.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956488+0000", 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4983000000000001e-05 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956673+0000", 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000184895 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956698+0000", 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4905999999999998e-05 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969407+0000", 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012709420000000001 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969740+0000", 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033311700000000002 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972472+0000", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027318680000000001 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.978868+0000", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063953710000000004 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.981617+0000", 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027492340000000001 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.981852+0000", 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023522500000000001 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 74/32 ECSubWrite(tid=114, reqid=osd.7.0:114, at_version=74'29, trim_to=0'0, roll_forward_to=73'27))", 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.967910+0000", 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.747436291, 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006776972, 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.967910+0000", 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.967910+0000", 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.967910+0000", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7599999999999997e-07 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.967964+0000", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4249999999999997e-05 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968075+0000", 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110746 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968098+0000", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2827e-05 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968240+0000", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014172499999999999 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968264+0000", 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4709999999999999e-05 2024-09-16T02:15:15.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.974594+0000", 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063300079999999998 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.974687+0000", 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2230000000000003e-05 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.es0 74/66 ECSubWriteReply(tid=69, last_complete=74'46, committed=1, applied=1))", 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.736233857, 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.58875802499999996, 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979114+0000", 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.643e-06 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979125+0000", 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1399999999999999e-05 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566505+0000", 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.58737945199999997 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566518+0000", 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.385e-05 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566681+0000", 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000162313 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.567870+0000", 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001189367 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 74/32 ECSubWrite(tid=94, reqid=osd.3.0:94, at_version=74'91, trim_to=0'0, roll_forward_to=73'87))", 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.727237945000001, 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.45024945, 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988110+0000", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6729999999999999e-06 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988147+0000", 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7766999999999999e-05 2024-09-16T02:15:15.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436677+0000", 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44852915199999999 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436695+0000", 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8009999999999999e-05 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436859+0000", 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016417499999999999 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436880+0000", 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0933000000000001e-05 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.438276+0000", 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013959269999999999 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.438357+0000", 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1812999999999997e-05 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:716 3.es0 3:7ab7d2d0:::smithi19530025-50:d [read 365291~1666325,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:08.032752+0000", 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.6825938990000004, 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070136180000000001, 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.766 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 716 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032752+0000", 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032752+0000", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032753+0000", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.407e-06 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032767+0000", 2024-09-16T02:15:15.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3388e-05 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032778+0000", 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0916e-05 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032787+0000", 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5030000000000003e-06 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032860+0000", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3025e-05 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032990+0000", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013007599999999999 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.039766+0000", 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067753029999999999 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.770 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:717 3.es0 3:7ab7d2d0:::smithi19530025-50:d [read 0~2031616,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:08.032805+0000", 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.6825413349999998, 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081421489999999996, 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.771 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 717 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032805+0000", 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032805+0000", 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032805+0000", 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5400000000000002e-07 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032809+0000", 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0999999999999997e-06 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032815+0000", 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2700000000000001e-06 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032819+0000", 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5499999999999999e-06 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033486+0000", 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066739999999999996 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033578+0000", 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2126000000000005e-05 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.040947+0000", 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073683489999999997 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:718 3.es0 3:7ab7d2d0:::smithi19530025-50:d [read 0~2031616,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.6825038570000004, 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092095790000000007, 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 718 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.776 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3599999999999999e-07 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032845+0000", 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9399999999999998e-06 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032856+0000", 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0881e-05 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032861+0000", 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5700000000000003e-06 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033868+0000", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00100712 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033940+0000", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1804000000000006e-05 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.042052+0000", 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081119279999999992 2024-09-16T02:15:15.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpReadReply(3.es0 84/83 ECSubReadReply(tid=86, attrs_read=0))", 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.5740697409999997, 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.77097236800000002, 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141277+0000", 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6700000000000002e-07 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141534+0000", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025689899999999999 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911063+0000", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76952898400000003 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911077+0000", 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4446e-05 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911215+0000", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013733300000000001 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.912249+0000", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010338389999999999 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 84/83 [PushReplyOp(3:7df442af:::smithi19530025-39:14)])", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:15.784 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.800843856, 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17568129900000001, 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914504+0000", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9599999999999999e-06 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914518+0000", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4306e-05 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089101+0000", 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17458259700000001 2024-09-16T02:15:15.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089122+0000", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1399999999999998e-05 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089292+0000", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016961600000000001 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090183+0000", 2024-09-16T02:15:15.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089141999999999997 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.bs4 85/83 ECSubWrite(tid=169, reqid=osd.7.0:169, at_version=85'52, trim_to=0'0, roll_forward_to=84'50))", 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.068808+0000", 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.6465378309999998, 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063241010000000004, 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068808+0000", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068808+0000", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068809+0000", 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9699999999999996e-07 2024-09-16T02:15:15.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068864+0000", 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5809e-05 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068919+0000", 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5009999999999997e-05 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.068936+0000", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6892999999999999e-05 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.069082+0000", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014609599999999999 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.069106+0000", 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3813000000000001e-05 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.075061+0000", 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059544510000000004 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.075132+0000", 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1432000000000003e-05 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 85/83 [PushReplyOp(3:7df442af:::smithi19530025-39:head)])", 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.6263097449999999, 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012806684, 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089038+0000", 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6360000000000001e-06 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089051+0000", 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3499999999999999e-05 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101196+0000", 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012145100000000001 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101215+0000", 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8496999999999999e-05 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101344+0000", 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128835 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101843+0000", 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049911599999999997 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 85/83 [PushReplyOp(3:7df442af:::smithi19530025-39:14)])", 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.101141+0000", 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.614204719, 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084058569999999992, 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101141+0000", 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101141+0000", 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101143+0000", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7770000000000001e-06 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101154+0000", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0986e-05 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107413+0000", 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006259092 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107428+0000", 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4956999999999999e-05 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107578+0000", 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000149655 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.109547+0000", 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019693900000000001 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 85/83 [PushReplyOp(3:7df442af:::smithi19530025-39:head)])", 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.60798653, 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42809435099999998, 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:15.801 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107361+0000", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.784e-06 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107372+0000", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.045e-05 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535025+0000", 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42765297299999999 2024-09-16T02:15:15.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535042+0000", 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7767000000000001e-05 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535170+0000", 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128119 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535454+0000", 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028325799999999999 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:767 3.es0 3:7df442af:::smithi19530025-39:head [copy-from ver 49] snapc 1a=[1a,18,d] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.64100790299999999, 2024-09-16T02:15:15.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026046024000000001, 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 767 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074340+0000", 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3700000000000002e-06 2024-09-16T02:15:15.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074350+0000", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7650000000000005e-06 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074371+0000", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0914999999999999e-05 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074380+0000", 2024-09-16T02:15:15.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1060000000000006e-06 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074459+0000", 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8447000000000003e-05 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074466+0000", 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2439999999999996e-06 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076762+0000", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022956880000000002 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076829+0000", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7267999999999994e-05 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.093614+0000", 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016785255999999998 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.096089+0000", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024751349999999998 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.100271+0000", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041815380000000003 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.100384+0000", 2024-09-16T02:15:15.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113292 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:768 3.es0 3:7df442af:::smithi19530025-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.64094737899999998, 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021202618999999999, 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:15.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 768 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0499999999999999e-07 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074401+0000", 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9379999999999999e-06 2024-09-16T02:15:15.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074416+0000", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4816e-05 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074422+0000", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4420000000000003e-06 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074523+0000", 2024-09-16T02:15:15.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100993 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074529+0000", 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2899999999999999e-06 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076959+0000", 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024291550000000001 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:15.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.077007+0000", 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8551000000000003e-05 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095460+0000", 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018452402 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095525+0000", 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5692999999999996e-05 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:15.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095601+0000", 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5933999999999995e-05 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:15.818 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:15.820 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:15.814+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:15.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:15.831 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:15.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:15.856 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:15.883 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:15.883 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:15.909 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:15.916 INFO:tasks.rados.rados.0.smithi195.stdout:306: done (1 left) 2024-09-16T02:15:15.916 INFO:tasks.rados.rados.0.smithi195.stdout:307: done (0 left) 2024-09-16T02:15:15.916 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 37 current snap is 25 2024-09-16T02:15:15.917 INFO:tasks.rados.rados.0.smithi195.stdout:308: seq_num 121 ranges {1818624=491520,2310144=802816,3112960=720896,3833856=32768} 2024-09-16T02:15:15.921 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:15.918+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:15.933 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:15.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:15.952 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:15.952 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:15.952 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:15.952 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:15.954 INFO:tasks.rados.rados.0.smithi195.stdout:308: writing smithi19530025-37 from 1818624 to 2310144 tid 1 2024-09-16T02:15:15.959 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:15.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:15.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:15.969 INFO:tasks.rados.rados.0.smithi195.stdout:308: writing smithi19530025-37 from 2310144 to 3112960 tid 2 2024-09-16T02:15:15.978 INFO:tasks.rados.rados.0.smithi195.stdout:308: writing smithi19530025-37 from 3112960 to 3833856 tid 3 2024-09-16T02:15:15.979 INFO:tasks.rados.rados.0.smithi195.stdout:308: writing smithi19530025-37 from 3833856 to 3866624 tid 4 2024-09-16T02:15:15.979 INFO:tasks.rados.rados.0.smithi195.stdout:309: copy_from oid 44 from oid 18 current snap is 25 2024-09-16T02:15:15.979 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 1 to smithi19530025-37 2024-09-16T02:15:15.979 INFO:tasks.rados.rados.0.smithi195.stdout:310: snap_create 2024-09-16T02:15:15.987 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 2 to smithi19530025-37 2024-09-16T02:15:15.987 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:15.987 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:15.988 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 3 to smithi19530025-37 2024-09-16T02:15:15.990 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 4 to smithi19530025-37 2024-09-16T02:15:16.002 INFO:tasks.rados.rados.0.smithi195.stdout:309: finishing copy_from racing read to smithi19530025-44 2024-09-16T02:15:16.014 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:16.014 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:16.020 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:16.014+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:16.035 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:16.035 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:16.060 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:16.060 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:16.061 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:16.090 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:16.091 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:16.113 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:16.113 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:16.113 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:16.115 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:16.115 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:16.122 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:16.118+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:16.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:16.146 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:16.147 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:16.172 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:16.172 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:16.182 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 5 to smithi19530025-37 2024-09-16T02:15:16.185 INFO:tasks.rados.rados.0.smithi195.stdout:308: finishing write tid 6 to smithi19530025-37 2024-09-16T02:15:16.185 INFO:tasks.rados.rados.0.smithi195.stdout:308: oid 37 updating version 0 to 88 2024-09-16T02:15:16.185 INFO:tasks.rados.rados.0.smithi195.stdout:308: oid 37 updating version 88 to 90 2024-09-16T02:15:16.185 INFO:tasks.rados.rados.0.smithi195.stdout:308: oid 37 version 90 is already newer than 87 2024-09-16T02:15:16.185 INFO:tasks.rados.rados.0.smithi195.stdout:308: oid 37 version 90 is already newer than 89 2024-09-16T02:15:16.186 INFO:tasks.rados.rados.0.smithi195.stdout:308: oid 37 version 90 is already newer than 86 2024-09-16T02:15:16.186 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 90 (ObjNum 121 snap 25 seq_num 121) dirty exists 2024-09-16T02:15:16.186 INFO:tasks.rados.rados.0.smithi195.stdout:308: left oid 37 (ObjNum 121 snap 25 seq_num 121) 2024-09-16T02:15:16.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:16.198+0000 7f6265d9b640 0 -- 172.21.15.3:0/2812106123 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f6268221a90 msgr2=0x7f6268223e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:16.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:16.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:16.258 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:16.254+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:16.276 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:16.276 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:16.291 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:16.339 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:16.337+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:16.421 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:16.421 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:16.424 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:16.421+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:16.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:16.524 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:16.522+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:16.602 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:16.602 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:16.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:16.625 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:16.622+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:16.691 INFO:tasks.rados.rados.0.smithi195.stdout:309: finishing copy_from to smithi19530025-44 2024-09-16T02:15:16.692 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 4 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T02:15:16.726 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:16.722+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:16.737 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:16.738 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:16.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:16.827 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:16.822+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:16.875 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:16.876 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:16.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:16.923 INFO:tasks.rados.rados.0.smithi195.stdout:308: done (2 left) 2024-09-16T02:15:16.923 INFO:tasks.rados.rados.0.smithi195.stdout:309: done (1 left) 2024-09-16T02:15:16.923 INFO:tasks.rados.rados.0.smithi195.stdout:310: done (0 left) 2024-09-16T02:15:16.923 INFO:tasks.rados.rados.0.smithi195.stdout:311: snap_remove snap 23 2024-09-16T02:15:16.927 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:16.926+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:17.014 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.014 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.028 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:17.026+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:17.029 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:17.129 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:17.126+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:17.152 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.152 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:17.217 INFO:teuthology.orchestra.run.smithi177.stdout:get_pg_num_history pg_num_history pg_num_history(e81 pg_nums {1={23=1},2={25=8},3={32=16}} deleted_pools ) 2024-09-16T02:15:17.217 INFO:teuthology.orchestra.run.smithi177.stdout:Importing pgid 3.3s6 2024-09-16T02:15:17.217 INFO:teuthology.orchestra.run.smithi177.stdout:Write 6#3:c481b30a:::smithi19530025-1:head# 2024-09-16T02:15:17.217 INFO:teuthology.orchestra.run.smithi177.stdout:snapset 6=[]:{} 2024-09-16T02:15:17.218 INFO:teuthology.orchestra.run.smithi177.stdout:Write 6#3:c628ba0c:::smithi19530025-27:head# 2024-09-16T02:15:17.218 INFO:teuthology.orchestra.run.smithi177.stdout:snapset 14=[]:{} 2024-09-16T02:15:17.219 INFO:teuthology.orchestra.run.smithi177.stdout:write_pg epoch 79 info 3.3s6( v 72'21 (0'0,72'21] local-lis/les=66/67 n=2 ec=32/32 lis/c=66/66 les/c/f=67/67/0 sis=66) 2024-09-16T02:15:17.220 INFO:teuthology.orchestra.run.smithi177.stdout:Import successful 2024-09-16T02:15:17.230 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:17.226+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:17.285 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.285 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:17.331 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:17.330+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:17.417 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.417 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:17.432 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:17.430+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:17.532 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:17.529+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:17.551 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.552 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:17.633 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:17.630+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:17.725 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.725 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.733 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:17.730+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:17.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:17.769 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /var/log/ceph/exp.3.3s6.10 2024-09-16T02:15:17.826 INFO:tasks.thrashosds.thrasher:checking if target objectstore is bluestore on osd.10 2024-09-16T02:15:17.826 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-label 2024-09-16T02:15:17.858 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.858 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:17.869 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:17.866+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:17.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:17.903 INFO:teuthology.orchestra.run.smithi177.stdout:inferring bluefs devices from bluestore path 2024-09-16T02:15:17.927 INFO:tasks.rados.rados.0.smithi195.stdout:311: done (0 left) 2024-09-16T02:15:17.927 INFO:tasks.rados.rados.0.smithi195.stdout:312: delete oid 41 current snap is 26 2024-09-16T02:15:17.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:15:17.935 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:17.933+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:17.940 INFO:tasks.rados.rados.0.smithi195.stdout:312: done (0 left) 2024-09-16T02:15:17.940 INFO:tasks.rados.rados.0.smithi195.stdout:313: snap_create 2024-09-16T02:15:17.996 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:17.996 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:18.010 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:18.036 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:18.030+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:18.137 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:18.130+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:18.142 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:18.142 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:18.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:18.206 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:18.206 INFO:teuthology.orchestra.run.smithi177.stdout: "/var/lib/ceph/osd/ceph-10/block": { 2024-09-16T02:15:18.206 INFO:teuthology.orchestra.run.smithi177.stdout: "osd_uuid": "99005d53-e62d-4c20-8d3c-541fd20cbeb8", 2024-09-16T02:15:18.206 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 96636764160, 2024-09-16T02:15:18.206 INFO:teuthology.orchestra.run.smithi177.stdout: "btime": "2024-09-16T02:13:22.254404+0000", 2024-09-16T02:15:18.207 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "main", 2024-09-16T02:15:18.207 INFO:teuthology.orchestra.run.smithi177.stdout: "bfm_blocks": "23592960", 2024-09-16T02:15:18.207 INFO:teuthology.orchestra.run.smithi177.stdout: "bfm_blocks_per_key": "128", 2024-09-16T02:15:18.207 INFO:teuthology.orchestra.run.smithi177.stdout: "bfm_bytes_per_block": "4096", 2024-09-16T02:15:18.207 INFO:teuthology.orchestra.run.smithi177.stdout: "bfm_size": "96636764160", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "bluefs": "1", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_fsid": "babbcdda-abe3-4b2f-82b4-da47bc408d34", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version_when_created": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "created_at": "2024-09-16T02:13:24.836542Z", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "elastic_shared_blobs": "1", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "epoch": "16", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "kv_backend": "rocksdb", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "magic": "ceph osd volume v026", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "multi": "yes", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "ready": "ready", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "require_osd_release": "19", 2024-09-16T02:15:18.208 INFO:teuthology.orchestra.run.smithi177.stdout: "type": "bluestore", 2024-09-16T02:15:18.209 INFO:teuthology.orchestra.run.smithi177.stdout: "whoami": "10" 2024-09-16T02:15:18.209 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.209 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:18.209 INFO:tasks.thrashosds.thrasher:checking if target bluestore supports sharding on osd.10 2024-09-16T02:15:18.210 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 show-sharding 2024-09-16T02:15:18.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:18.238 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:18.233+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:18.282 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:18.282 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:18.296 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:18.338 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:18.333+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:18.377 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:18.377 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:18.403 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:18.403 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:18.416 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:18.416 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:18.431 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:18.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:18.432 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:18.439 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:18.437+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:18.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:18.462 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:18.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:18.495 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:18.527 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:18.527 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:18.540 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:18.533+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:18.555 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:18.556 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:18.557 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:18.557 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:18.557 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:18.557 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:18.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:18.589 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:18.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:18.619 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:18.619 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:18.641 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:18.637+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:18.646 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:18.646 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:18.680 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:18.680 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:18.711 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:18.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:18.729 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:18.729 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:18.729 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:18.729 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:18.729 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:18.742 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:18.738+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:18.744 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:18.745 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:18.745 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:18.777 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:18.777 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:18.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:18.782+0000 7efe6bfff640 0 -- 172.21.15.3:0/1654829415 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7efe6c222310 msgr2=0x7efe6c224760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:18.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:18.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:18.827 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:18.842 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:18.838+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.872 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:196 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.548842579999999, 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013336756999999999, 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 196 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.873 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321914+0000", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321917+0000", 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6970000000000002e-06 2024-09-16T02:15:18.874 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321944+0000", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7226e-05 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.321985+0000", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1551000000000002e-05 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322009+0000", 2024-09-16T02:15:18.875 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3269e-05 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322176+0000", 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016762 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322357+0000", 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000180805 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.876 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.324174+0000", 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001817113 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330068+0000", 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058941790000000003 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335031+0000", 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049623899999999997 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.877 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335251+0000", 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021990700000000001 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:197 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.548699878999997, 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013892593, 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.878 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.879 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322058+0000", 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.643e-06 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322069+0000", 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.044e-05 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.880 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322090+0000", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1455e-05 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322106+0000", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6163000000000001e-05 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325364+0000", 2024-09-16T02:15:18.881 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032580809999999999 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325616+0000", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025181900000000002 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327103+0000", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014866549999999999 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330072+0000", 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029688729999999999 2024-09-16T02:15:18.882 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335817+0000", 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057450959999999999 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335949+0000", 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132368 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.883 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:198 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.548605486, 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014215244, 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 198 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.884 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322152+0000", 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3829999999999999e-06 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.885 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322159+0000", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5660000000000003e-06 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322179+0000", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9613e-05 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322190+0000", 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1542000000000001e-05 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.886 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328227+0000", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060372159999999998 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328485+0000", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025810200000000002 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.329430+0000", 2024-09-16T02:15:18.887 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094416499999999996 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330786+0000", 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013565509999999999 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336208+0000", 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054220529999999996 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.888 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336366+0000", 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015805300000000001 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 44/32 ECSubWrite(tid=43, reqid=osd.11.0:43, at_version=44'35, trim_to=0'0, roll_forward_to=43'31))", 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.520209231999999, 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010145988, 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.889 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350547+0000", 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.890 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350559+0000", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1813e-05 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350610+0000", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1325000000000001e-05 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350717+0000", 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106846 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.891 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.350739+0000", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2181999999999999e-05 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352445+0000", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017059110000000001 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352470+0000", 2024-09-16T02:15:18.892 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4185999999999998e-05 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.360599+0000", 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0081295510000000005 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.360693+0000", 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4173999999999997e-05 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.893 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.510607731, 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082887169999999993, 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.894 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360149+0000", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360150+0000", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9000000000000005e-07 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360200+0000", 2024-09-16T02:15:18.895 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.9781000000000003e-05 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360340+0000", 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014055900000000001 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360360+0000", 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0205999999999999e-05 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.896 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360530+0000", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000169414 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.360554+0000", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3828999999999998e-05 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.368346+0000", 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.007792902 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.897 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.368438+0000", 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1036e-05 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.50893465, 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095998799999999999, 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.898 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361822+0000", 2024-09-16T02:15:18.899 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361897+0000", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.4048000000000007e-05 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361976+0000", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9432999999999997e-05 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361998+0000", 2024-09-16T02:15:18.900 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1996000000000001e-05 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362235+0000", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023705199999999999 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362259+0000", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3782e-05 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371349+0000", 2024-09-16T02:15:18.901 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090897349999999998 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371422+0000", 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.3257999999999996e-05 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 45/32 ECSubWrite(tid=194, reqid=osd.13.0:194, at_version=45'30, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.508076258000003, 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086860159999999995, 2024-09-16T02:15:18.902 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362680+0000", 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.903 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362682+0000", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0839999999999999e-06 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362739+0000", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6721000000000001e-05 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362788+0000", 2024-09-16T02:15:18.904 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8875000000000002e-05 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362811+0000", 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3275e-05 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362968+0000", 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015707199999999999 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.905 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.362993+0000", 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4865999999999999e-05 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371258+0000", 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082643939999999996 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.906 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.371366+0000", 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108729 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 44.505898999000003, 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091943479999999998, 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.907 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364858+0000", 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.908 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.364860+0000", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1169999999999998e-06 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365009+0000", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000149808 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365058+0000", 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8677999999999997e-05 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.909 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365081+0000", 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2486000000000001e-05 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365254+0000", 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017369400000000001 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.910 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.365280+0000", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5505000000000001e-05 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.373957+0000", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00867712 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.374052+0000", 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4939999999999996e-05 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.911 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 57/32 ECSubWrite(tid=117, reqid=osd.11.0:117, at_version=57'104, trim_to=0'0, roll_forward_to=56'100))", 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 32.938480859000002, 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013673405, 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.912 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932276+0000", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932278+0000", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9439999999999999e-06 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932320+0000", 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2052000000000003e-05 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.913 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932390+0000", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.0724999999999998e-05 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932406+0000", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932582+0000", 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000176632 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.914 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.932609+0000", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6389000000000001e-05 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.943840+0000", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011230795 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:45.945949+0000", 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021094719999999998 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.915 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:494 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.913693522999999, 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020145202000000001, 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 494 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.916 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957066+0000", 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0340000000000001e-06 2024-09-16T02:15:18.917 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957080+0000", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3696000000000001e-05 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957110+0000", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0252000000000001e-05 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957135+0000", 2024-09-16T02:15:18.918 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4586000000000001e-05 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957642+0000", 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050762300000000002 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957870+0000", 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022741699999999999 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.919 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.960012+0000", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002142793 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966293+0000", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062808969999999997 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977023+0000", 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010729446 2024-09-16T02:15:18.920 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977208+0000", 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000185458 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:495 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback d] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.913573746000001, 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021622549000000001, 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.921 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 495 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.922 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957185+0000", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957191+0000", 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.923 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957211+0000", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0069e-05 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957229+0000", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7856e-05 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961650+0000", 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044206920000000004 2024-09-16T02:15:18.924 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961911+0000", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026131599999999999 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.963434+0000", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00152287 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966303+0000", 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028691599999999999 2024-09-16T02:15:18.925 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978597+0000", 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012293652 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978805+0000", 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000208304 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.926 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:496 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 31.913471221999998, 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021768414, 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 496 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.927 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.928 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957287+0000", 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957294+0000", 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7800000000000003e-06 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957313+0000", 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9669000000000001e-05 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.929 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957328+0000", 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4847e-05 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964542+0000", 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0072143199999999998 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.930 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964700+0000", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157343 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.965510+0000", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080991199999999998 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.968798+0000", 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032880489999999999 2024-09-16T02:15:18.931 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978963+0000", 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010165014 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.979054+0000", 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1037000000000003e-05 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.932 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:583 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 0~3407872,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.961485044, 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011937642, 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 583 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.933 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909271+0000", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909274+0000", 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8499999999999998e-06 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.934 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2686e-05 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909317+0000", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9986999999999999e-05 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909339+0000", 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2305000000000001e-05 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.935 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909427+0000", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7862999999999995e-05 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909520+0000", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2571000000000005e-05 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.921209+0000", 2024-09-16T02:15:18.936 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011689379999999999 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:584 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 1157777~2250095,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.961366167000001, 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012758666, 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.937 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 584 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909390+0000", 2024-09-16T02:15:18.938 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909392+0000", 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9369999999999998e-06 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909401+0000", 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4300000000000006e-06 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.939 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909422+0000", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1022999999999998e-05 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909438+0000", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5739e-05 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909938+0000", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050038899999999996 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.940 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.910047+0000", 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010885199999999999 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.922149+0000", 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012102296 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:585 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 1560470~1847402,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 22.961291965000001, 2024-09-16T02:15:18.941 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013504089, 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 585 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.942 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909465+0000", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909466+0000", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3829999999999999e-06 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909477+0000", 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0567000000000001e-05 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.943 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909494+0000", 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7796e-05 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.909508+0000", 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3607e-05 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.944 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.912332+0000", 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028238 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.912425+0000", 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3457999999999999e-05 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:55.922969+0000", 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010543478 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.945 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 74/69 ECSubWrite(tid=137, reqid=osd.11.0:137, at_version=74'135, trim_to=0'0, roll_forward_to=73'131))", 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 15.890257663, 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.469769029, 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:18.946 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980500+0000", 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5030000000000001e-06 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.947 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980539+0000", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8754000000000002e-05 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448625+0000", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46808555400000001 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448640+0000", 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.948 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448730+0000", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9498000000000003e-05 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448742+0000", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2479e-05 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.450231+0000", 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014881320000000001 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.949 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.450268+0000", 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7307e-05 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:713 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 0~1818624,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.033272+0000", 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.837484751, 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012140695, 2024-09-16T02:15:18.950 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 713 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033272+0000", 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.951 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033272+0000", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033276+0000", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9199999999999997e-06 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033291+0000", 2024-09-16T02:15:18.952 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5608999999999999e-05 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033353+0000", 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1933000000000005e-05 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033377+0000", 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3272e-05 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.953 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033598+0000", 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000221552 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033873+0000", 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027452100000000001 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.045412+0000", 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011539888 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.954 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:714 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 947105~871519,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.837321223, 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013949832000000001, 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 714 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.955 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033437+0000", 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7129999999999999e-06 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.956 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033450+0000", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2782999999999999e-05 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033477+0000", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7560999999999999e-05 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033493+0000", 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5133000000000001e-05 2024-09-16T02:15:18.957 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.036275+0000", 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027821389999999999 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.036467+0000", 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019283200000000001 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.958 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.047385+0000", 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010917671 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:715 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 328282~1490342,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:18.959 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.837164229000001, 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01542255, 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 715 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.960 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033594+0000", 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.154e-06 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.961 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033608+0000", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033644+0000", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5831999999999998e-05 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033665+0000", 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1121999999999999e-05 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.962 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041530+0000", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078650109999999999 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041711+0000", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018134999999999999 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.049015+0000", 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0073036059999999998 2024-09-16T02:15:18.963 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:770 3.ds0 3:ba16c3d6:::smithi19530025-37:head [append 0~491520] snapc 1b=[1b,1a,18,d] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 2.9154375510000001, 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021836291000000001, 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:18.964 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 770 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:18.965 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955325+0000", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.604e-06 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956113+0000", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00078869400000000005 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956132+0000", 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8702000000000002e-05 2024-09-16T02:15:18.966 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956142+0000", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956310+0000", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000167467 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956490+0000", 2024-09-16T02:15:18.967 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018010199999999999 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.958995+0000", 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025054489999999999 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.962085+0000", 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030896220000000002 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.968 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.977072+0000", 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014986963000000001 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.977155+0000", 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3232000000000005e-05 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:18.969 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:18.971 INFO:tasks.rados.rados.0.smithi195.stdout:313: done (0 left) 2024-09-16T02:15:18.971 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 48 current snap is 27 2024-09-16T02:15:18.971 INFO:tasks.rados.rados.0.smithi195.stdout:314: seq_num 123 ranges {1572864=524288,2097152=573440,2670592=688128,3358720=294912} 2024-09-16T02:15:18.971 INFO:tasks.rados.rados.0.smithi195.stdout:314: writing smithi19530025-48 from 1572864 to 2097152 tid 1 2024-09-16T02:15:18.972 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:18.966+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:18.972 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:15:18.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:18.980 INFO:tasks.rados.rados.0.smithi195.stdout:314: writing smithi19530025-48 from 2097152 to 2670592 tid 2 2024-09-16T02:15:18.988 INFO:tasks.rados.rados.0.smithi195.stdout:314: writing smithi19530025-48 from 2670592 to 3358720 tid 3 2024-09-16T02:15:18.991 INFO:tasks.rados.rados.0.smithi195.stdout:314: writing smithi19530025-48 from 3358720 to 3653632 tid 4 2024-09-16T02:15:18.991 INFO:tasks.rados.rados.0.smithi195.stdout:315: setattr oid 32 current snap is 27 2024-09-16T02:15:18.991 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 1 to smithi19530025-48 2024-09-16T02:15:18.991 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 2 to smithi19530025-48 2024-09-16T02:15:18.995 INFO:tasks.rados.rados.0.smithi195.stdout:316: rollback oid 21 current snap is 27 2024-09-16T02:15:18.995 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 21 to 25 2024-09-16T02:15:18.995 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 15 current snap is 27 2024-09-16T02:15:18.995 INFO:tasks.rados.rados.0.smithi195.stdout:317: seq_num 124 ranges {245760=540672,786432=720896,1507328=786432,2293760=49152} 2024-09-16T02:15:19.004 INFO:tasks.rados.rados.0.smithi195.stdout:317: writing smithi19530025-15 from 245760 to 786432 tid 1 2024-09-16T02:15:19.012 INFO:tasks.rados.rados.0.smithi195.stdout:317: writing smithi19530025-15 from 786432 to 1507328 tid 2 2024-09-16T02:15:19.020 INFO:tasks.rados.rados.0.smithi195.stdout:317: writing smithi19530025-15 from 1507328 to 2293760 tid 3 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:317: writing smithi19530025-15 from 2293760 to 2342912 tid 4 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:318: read oid 33 snap -1 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:318: expect deleted 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:319: rollback oid 3 current snap is 27 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 3 to smithi19530025-48 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 4 to smithi19530025-48 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 32 v 116 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:316: finishing rollback tid 0 to smithi19530025-21 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:316: finishing rollback tid 1 to smithi19530025-21 2024-09-16T02:15:19.021 INFO:tasks.rados.rados.0.smithi195.stdout:316: finishing rollback tid 2 to smithi19530025-21 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 21 v 65 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 3 to 21 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:315: done (5 left) 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:316: done (4 left) 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 4 current snap is 27 2024-09-16T02:15:19.022 INFO:tasks.rados.rados.0.smithi195.stdout:320: seq_num 125 ranges {0=557056,557056=475136,1032192=507904,1540096=475136,2015232=98304} 2024-09-16T02:15:19.025 INFO:tasks.rados.rados.0.smithi195.stdout:320: writing smithi19530025-4 from 0 to 557056 tid 1 2024-09-16T02:15:19.029 INFO:tasks.rados.rados.0.smithi195.stdout:320: writing smithi19530025-4 from 557056 to 1032192 tid 2 2024-09-16T02:15:19.033 INFO:tasks.rados.rados.0.smithi195.stdout:320: writing smithi19530025-4 from 1032192 to 1540096 tid 3 2024-09-16T02:15:19.036 INFO:tasks.rados.rados.0.smithi195.stdout:320: writing smithi19530025-4 from 1540096 to 2015232 tid 4 2024-09-16T02:15:19.037 INFO:tasks.rados.rados.0.smithi195.stdout:320: writing smithi19530025-4 from 2015232 to 2113536 tid 5 2024-09-16T02:15:19.037 INFO:tasks.rados.rados.0.smithi195.stdout:321: snap_remove snap 24 2024-09-16T02:15:19.072 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:19.070+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:19.104 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:19.104 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:19.104 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:19.105 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:19.119 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:19.172 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:19.165+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:19.242 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:19.243 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:19.243 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:19.243 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:19.243 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:19.258 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:19.274 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:19.268+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:19.373 INFO:teuthology.orchestra.run.smithi177.stdout:m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32 2024-09-16T02:15:19.374 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:19.368+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:19.378 INFO:tasks.thrashosds.thrasher:applying new sharding to bluestore on osd.10 2024-09-16T02:15:19.378 INFO:tasks.thrashosds.thrasher:generate random sharding 2024-09-16T02:15:19.378 INFO:tasks.thrashosds.thrasher:applying new sharding: m P(2) 2024-09-16T02:15:19.378 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 --sharding 'm P(2)' reshard 2024-09-16T02:15:19.424 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:19.424 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:19.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:19.424 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:19.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=114, reqid=client.4598.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.129128072, 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014358203, 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294123+0000", 2024-09-16T02:15:19.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157742 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294233+0000", 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109842 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294256+0000", 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3779000000000001e-05 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294602+0000", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000345817 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294625+0000", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308228+0000", 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013602725 2024-09-16T02:15:19.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308323+0000", 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4631999999999998e-05 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=115, reqid=client.4598.0:29, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:15:19.429 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.125352200999998, 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010657378, 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297740+0000", 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297742+0000", 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297835+0000", 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2315e-05 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298241+0000", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040658699999999997 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298273+0000", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1566000000000002e-05 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298698+0000", 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00042513000000000002 2024-09-16T02:15:19.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298724+0000", 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5681999999999999e-05 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308331+0000", 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096073479999999999 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308398+0000", 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.6749999999999996e-05 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.125191805999997, 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013936119, 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.434 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7299999999999996e-07 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298091+0000", 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000189612 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298361+0000", 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026983699999999998 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298377+0000", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6619000000000001e-05 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298580+0000", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020297699999999999 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298602+0000", 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1823000000000001e-05 2024-09-16T02:15:19.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311758+0000", 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013155635000000001 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311837+0000", 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9042999999999995e-05 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.124994698000002, 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013796841000000001, 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:19.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298099+0000", 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2999999999999999e-07 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298132+0000", 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3460999999999997e-05 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298437+0000", 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000305149 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298455+0000", 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8099000000000001e-05 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299840+0000", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001384826 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5151999999999999e-05 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311844+0000", 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011978284000000001 2024-09-16T02:15:19.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311895+0000", 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1039999999999999e-05 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.5s3 35/32 ECSubWrite(tid=2, reqid=client.4598.0:13, at_version=35'3, trim_to=0'0, roll_forward_to=34'1))", 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.124207431999999, 2024-09-16T02:15:19.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011300899, 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298885+0000", 2024-09-16T02:15:19.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298887+0000", 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5400000000000001e-06 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298992+0000", 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000105472 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299113+0000", 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012086099999999999 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299135+0000", 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1679e-05 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299294+0000", 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015895199999999999 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299322+0000", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8246e-05 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310061+0000", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010739165 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310186+0000", 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012498399999999999 2024-09-16T02:15:19.447 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.121916489, 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010788813, 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301176+0000", 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301178+0000", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.77e-06 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301420+0000", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024188200000000001 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301595+0000", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017550199999999999 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301613+0000", 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7391999999999999e-05 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302202+0000", 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000588903 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302225+0000", 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311905+0000", 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0096801509999999997 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311965+0000", 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9593999999999999e-05 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.452 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.121666028, 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013049734, 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:19.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6052000000000001e-05 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000240423 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301716+0000", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2289e-05 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305459+0000", 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037425710000000001 2024-09-16T02:15:19.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305488+0000", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8756e-05 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314378+0000", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088900869999999996 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314476+0000", 2024-09-16T02:15:19.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8669e-05 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.121279891999997, 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014210883000000001, 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.457 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301814+0000", 2024-09-16T02:15:19.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4130000000000001e-06 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302022+0000", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000207844 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302120+0000", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7864999999999999e-05 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302140+0000", 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0156000000000001e-05 2024-09-16T02:15:19.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310588+0000", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084482120000000001 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310612+0000", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3879e-05 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315921+0000", 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053087079999999997 2024-09-16T02:15:19.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316024+0000", 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102806 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.120268512999999, 2024-09-16T02:15:19.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013263980999999999, 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:19.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302826+0000", 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9240000000000001e-06 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302901+0000", 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5172999999999993e-05 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.463 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304024+0000", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001123181 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304072+0000", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8127999999999997e-05 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312045+0000", 2024-09-16T02:15:19.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079722669999999999 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312071+0000", 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6291999999999998e-05 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316033+0000", 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039623380000000001 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316088+0000", 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4678000000000001e-05 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:49 3.cs0 3:3203a6be:::smithi19530025-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.107320481000002, 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016954778, 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.466 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315781+0000", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8699999999999998e-06 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316702+0000", 2024-09-16T02:15:19.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092113000000000004 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316734+0000", 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2375000000000001e-05 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316756+0000", 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1843000000000001e-05 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317070+0000", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031399899999999997 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317385+0000", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031517099999999999 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.320926+0000", 2024-09-16T02:15:19.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035407450000000001 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332340+0000", 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011414260000000001 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332532+0000", 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191137 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332727+0000", 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000195248 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:50 3.cs0 3:3203a6be:::smithi19530025-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.106279698999998, 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017275975999999998, 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:19.472 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316815+0000", 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0899999999999999e-06 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316826+0000", 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1242e-05 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316851+0000", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4969000000000001e-05 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316868+0000", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6535999999999999e-05 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324308+0000", 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074406289999999998 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324542+0000", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023414200000000001 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.325642+0000", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001099182 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332739+0000", 2024-09-16T02:15:19.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070972090000000002 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.333048+0000", 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030949200000000002 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334089+0000", 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010404850000000001 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.477 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:51 3.cs0 3:3203a6be:::smithi19530025-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.106194203000001, 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019761212, 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.478 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 51 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316900+0000", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.477e-06 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316907+0000", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8889999999999996e-06 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316927+0000", 2024-09-16T02:15:19.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0591999999999999e-05 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316943+0000", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5756999999999998e-05 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327066+0000", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010123037 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327269+0000", 2024-09-16T02:15:19.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309000000000001 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334254+0000", 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069844 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334498+0000", 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024477499999999997 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.482 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336660+0000", 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021611949999999999 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/32 ECSubWrite(tid=224, reqid=osd.13.0:224, at_version=51'45, trim_to=0'0, roll_forward_to=49'43))", 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.008177680999999, 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24539247, 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.483 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2000000000000001e-07 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414963+0000", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8265e-05 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658566+0000", 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24360277699999999 2024-09-16T02:15:19.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658581+0000", 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4253000000000001e-05 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658749+0000", 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000168691 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658768+0000", 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8332999999999999e-05 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660236+0000", 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014682129999999999 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660307+0000", 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1618000000000004e-05 2024-09-16T02:15:19.487 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.1s1 62/61 ECSubWriteReply(tid=22, last_complete=62'113, committed=1, applied=1))", 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 28.391529029000001, 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66094662900000001, 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.488 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2300000000000002e-07 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031567+0000", 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3830000000000001e-06 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690362+0000", 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.65879452900000002 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690376+0000", 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.397e-05 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690498+0000", 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012224 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.692510+0000", 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020120839999999999 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:525 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.731510611000001, 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014305174, 2024-09-16T02:15:19.492 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 525 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691585+0000", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.667e-06 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691596+0000", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1837e-05 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691625+0000", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8714999999999999e-05 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691640+0000", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5027e-05 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.693656+0000", 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020154190000000001 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.694058+0000", 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040255 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.705887+0000", 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011828959 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.496 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:526 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.731424653000001, 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018909656, 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 526 2024-09-16T02:15:19.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691669+0000", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.19e-06 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691676+0000", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.277e-06 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691700+0000", 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3842e-05 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691718+0000", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.823e-05 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695237+0000", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035182619999999999 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695450+0000", 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021319899999999999 2024-09-16T02:15:19.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.710578+0000", 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015127656 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:527 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.731349258000002, 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023766461999999999, 2024-09-16T02:15:19.501 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 527 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691744+0000", 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2299999999999999e-07 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691749+0000", 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0370000000000002e-06 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691763+0000", 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.342e-05 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691770+0000", 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5320000000000001e-06 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.697667+0000", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058967209999999997 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.698047+0000", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000379768 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715510+0000", 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017463060999999998 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.505 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpReadReply(3.1s1 62/61 ECSubReadReply(tid=47, attrs_read=0))", 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.720759197, 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013250526, 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:19.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702334+0000", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.074e-06 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702822+0000", 2024-09-16T02:15:19.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048715800000000001 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702872+0000", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0565000000000001e-05 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702881+0000", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.49e-06 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.711121+0000", 2024-09-16T02:15:19.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082399199999999995 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715584+0000", 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044633190000000003 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1s1 62/61 [PushReplyOp(3:830a71a8:::smithi19530025-14:head)])", 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:19.509 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.718879043000001, 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19260793200000001, 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1399999999999997e-07 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704220+0000", 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1759999999999998e-06 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894917+0000", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19069640900000001 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894932+0000", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5690000000000001e-05 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.895110+0000", 2024-09-16T02:15:19.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000177546 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.896821+0000", 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017116970000000001 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 73/32 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 17.468660998000001, 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35829697500000002, 2024-09-16T02:15:19.513 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954434+0000", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.933e-06 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954491+0000", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7326e-05 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310497+0000", 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35600593200000002 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310519+0000", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2121999999999999e-05 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310700+0000", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018157399999999999 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310716+0000", 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5696e-05 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.312660+0000", 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001944274 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.312729+0000", 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8118000000000001e-05 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:19.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:19.518 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:19.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:19.520 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:19.518+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:19.620 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:19.613+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:19.651 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:19.651 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:19.651 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:19.651 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:19.665 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:19.721 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:19.718+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:19.787 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:19.787 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:19.787 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:19.788 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:19.788 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:19.802 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:19.848 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:19.846+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:19.922 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:19.918+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 52.587849200000001, 2024-09-16T02:15:19.927 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.334730205, 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:19.928 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335879+0000", 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335907+0000", 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7291999999999999e-05 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.929 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669030+0000", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33312349400000002 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669045+0000", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4943000000000001e-05 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669245+0000", 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019942899999999999 2024-09-16T02:15:19.930 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669263+0000", 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8445999999999999e-05 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670533+0000", 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001270024 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.931 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670602+0000", 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8406999999999997e-05 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.588876394000003, 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75881597999999995, 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.932 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.933 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334845+0000", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0900000000000001e-07 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334886+0000", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1254999999999997e-05 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091945+0000", 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75705888700000001 2024-09-16T02:15:19.934 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091960+0000", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4686e-05 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092157+0000", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019717000000000001 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092176+0000", 2024-09-16T02:15:19.935 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9020000000000001e-05 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093578+0000", 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001401769 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093660+0000", 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2484000000000005e-05 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.936 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.571853296, 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017058792, 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:19.937 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351868+0000", 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.938 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351917+0000", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352051+0000", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013457599999999999 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352073+0000", 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1489e-05 2024-09-16T02:15:19.939 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352231+0000", 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158455 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352260+0000", 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8472000000000001e-05 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.940 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368854+0000", 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016594647000000001 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368926+0000", 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1534999999999999e-05 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=35, last_complete=45'32, committed=1, applied=1))", 2024-09-16T02:15:19.941 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.541939272, 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65850650099999997, 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:19.942 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381783+0000", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.857e-06 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381795+0000", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1466e-05 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040081+0000", 2024-09-16T02:15:19.943 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65828677099999999 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040095+0000", 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3686000000000001e-05 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040215+0000", 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011949 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.944 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040288+0000", 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3231000000000004e-05 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.526733178999997, 2024-09-16T02:15:19.945 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27792197099999999, 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:19.946 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396988+0000", 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759999999999999e-06 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397010+0000", 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1525000000000001e-05 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.947 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673630+0000", 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27662027 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673655+0000", 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5003000000000001e-05 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673857+0000", 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020175299999999999 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.948 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673875+0000", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674832+0000", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095631999999999998 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674909+0000", 2024-09-16T02:15:19.949 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7741e-05 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:487 3.7s0 3:ed091bc8:::smithi19530025-23:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc d=[d,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.991107178, 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019918874, 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:19.950 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 487 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.951 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932617+0000", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.309e-06 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932695+0000", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8548999999999996e-05 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932736+0000", 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0349000000000003e-05 2024-09-16T02:15:19.952 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932757+0000", 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1903e-05 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932928+0000", 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017018600000000001 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:19.953 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932950+0000", 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2395000000000002e-05 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934568+0000", 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00161812 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934773+0000", 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000204995 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.954 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936578+0000", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001804709 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.940066+0000", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034883869999999999 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952401+0000", 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012334539 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.955 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952532+0000", 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131433 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.982705402000001, 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013412999, 2024-09-16T02:15:19.956 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:19.957 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941016+0000", 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6000000000000004e-07 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941065+0000", 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9462000000000002e-05 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.958 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941113+0000", 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941128+0000", 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4348999999999999e-05 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941254+0000", 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012600400000000001 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.959 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941277+0000", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3482999999999999e-05 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954333+0000", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013055430999999999 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954428+0000", 2024-09-16T02:15:19.960 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5468000000000001e-05 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=50, reqid=client.4598.0:494, at_version=58'42, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 32.964774456000001, 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010360967, 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.961 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.962 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958947+0000", 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4300000000000003e-07 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959048+0000", 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101671 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959143+0000", 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5120999999999999e-05 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.963 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959174+0000", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0803000000000003e-05 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959376+0000", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202105 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959404+0000", 2024-09-16T02:15:19.964 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7141999999999999e-05 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969241+0000", 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098377399999999993 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969307+0000", 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5741999999999995e-05 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.965 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 32.958042237000001, 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011275655000000001, 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.966 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965681+0000", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2199999999999999e-06 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965731+0000", 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0429000000000001e-05 2024-09-16T02:15:19.967 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965878+0000", 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014756 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965904+0000", 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5686000000000001e-05 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.968 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967710+0000", 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001805652 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.968220+0000", 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050980399999999999 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976848+0000", 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086280009999999997 2024-09-16T02:15:19.969 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976954+0000", 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106303 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=52, reqid=client.4598.0:496, at_version=58'44, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 32.957978195000003, 2024-09-16T02:15:19.970 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011274895, 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.971 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965743+0000", 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1069999999999999e-06 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965783+0000", 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9165999999999997e-05 2024-09-16T02:15:19.972 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967647+0000", 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018640340000000001 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967671+0000", 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4712e-05 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.973 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971326+0000", 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036543130000000002 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971353+0000", 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7376e-05 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976963+0000", 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056098989999999998 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.974 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.977017+0000", 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4287999999999999e-05 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=86, reqid=osd.6.0:86, at_version=73'70, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.978472814, 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015494396000000001, 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.975 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.976 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945254+0000", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3130000000000004e-06 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945315+0000", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0584999999999997e-05 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945408+0000", 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3288999999999994e-05 2024-09-16T02:15:19.977 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945425+0000", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7329e-05 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945587+0000", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016137399999999999 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945615+0000", 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8048999999999999e-05 2024-09-16T02:15:19.978 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959667+0000", 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014052007 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.960742+0000", 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010754499999999999 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.979 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=87, reqid=osd.6.0:87, at_version=73'72, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.975642948000001, 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020784545000000001, 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.980 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948079+0000", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.877e-06 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948132+0000", 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2312000000000003e-05 2024-09-16T02:15:19.981 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948237+0000", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105015 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948259+0000", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2668999999999999e-05 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948410+0000", 2024-09-16T02:15:19.982 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015014100000000001 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948436+0000", 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6060000000000001e-05 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968776+0000", 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020339946000000001 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.983 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968862+0000", 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6525000000000003e-05 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs5 73/32 ECSubWrite(tid=80, reqid=osd.14.0:80, at_version=73'29, trim_to=0'0, roll_forward_to=70'27))", 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.961000676000001, 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019480810000000001, 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.984 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962721+0000", 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-07 2024-09-16T02:15:19.985 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962770+0000", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9233000000000001e-05 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962872+0000", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010231800000000001 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962901+0000", 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8765999999999999e-05 2024-09-16T02:15:19.986 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963118+0000", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021719600000000001 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963156+0000", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7435999999999999e-05 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982116+0000", 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018960226 2024-09-16T02:15:19.987 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982201+0000", 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4818e-05 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 73/32 ECSubWrite(tid=76, reqid=osd.10.0:76, at_version=73'44, trim_to=0'0, roll_forward_to=71'42))", 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964218+0000", 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.959502867000001, 2024-09-16T02:15:19.988 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0096339770000000002, 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964218+0000", 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964218+0000", 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.989 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964218+0000", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9299999999999997e-07 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964270+0000", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1319000000000001e-05 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964379+0000", 2024-09-16T02:15:19.990 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000109088 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964402+0000", 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2753999999999999e-05 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968377+0000", 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039751020000000003 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.991 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968403+0000", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5819000000000001e-05 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973755+0000", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005352107 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973852+0000", 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6995000000000002e-05 2024-09-16T02:15:19.992 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=133, reqid=osd.11.0:133, at_version=73'129, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.957925254999999, 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016499698, 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.993 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.994 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.17e-07 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965821+0000", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.569e-05 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965896+0000", 2024-09-16T02:15:19.995 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5185999999999995e-05 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965907+0000", 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0379e-05 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966895+0000", 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098857000000000003 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.996 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966922+0000", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7066000000000001e-05 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982221+0000", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015298589 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982295+0000", 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4000999999999998e-05 2024-09-16T02:15:19.997 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=134, reqid=osd.11.0:134, at_version=73'131, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.957897321000001, 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017078139999999999, 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.998 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.03e-07 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:19.999 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965841+0000", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7326e-05 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965939+0000", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8420999999999998e-05 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965956+0000", 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6988999999999999e-05 2024-09-16T02:15:20.000 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972222+0000", 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062658890000000002 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972246+0000", 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4403e-05 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.001 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982860+0000", 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010613734 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982901+0000", 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1174999999999998e-05 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=88, reqid=osd.6.0:87, at_version=0'0, trim_to=0'0, roll_forward_to=73'72))", 2024-09-16T02:15:20.002 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.953053164, 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012896283, 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.003 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970668+0000", 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970682+0000", 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.322e-05 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.004 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970722+0000", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0429000000000002e-05 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970732+0000", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0105999999999999e-05 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.971118+0000", 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038623200000000002 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.005 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.971142+0000", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.983483+0000", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012341193 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.983564+0000", 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0289000000000003e-05 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.006 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 74/69 ECSubWrite(tid=137, reqid=osd.11.0:137, at_version=74'135, trim_to=0'0, roll_forward_to=73'131))", 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.944744925999998, 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010379599999999999, 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:20.007 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978977+0000", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8670000000000001e-06 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979030+0000", 2024-09-16T02:15:20.008 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2447000000000001e-05 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979134+0000", 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103835 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979156+0000", 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2558999999999999e-05 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.009 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979320+0000", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000163738 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979347+0000", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7282000000000001e-05 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.989264+0000", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099165069999999998 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.010 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.989355+0000", 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1365000000000004e-05 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:724 3.7s0 3:ebce791d:::smithi19530025-35:head [append 0~0] snapc 18=[18,d] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.9060592950000004, 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013022176999999999, 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.011 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 724 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:20.012 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017662+0000", 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9400000000000004e-07 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017666+0000", 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8999999999999999e-06 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.013 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017676+0000", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0386e-05 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017690+0000", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3502999999999999e-05 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.024290+0000", 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006600318 2024-09-16T02:15:20.014 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.024505+0000", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021444500000000001 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.025313+0000", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00080815300000000004 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.027220+0000", 2024-09-16T02:15:20.015 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001907066 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.030639+0000", 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034191949999999999 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.030683+0000", 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4416999999999998e-05 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.016 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:788 3.7s0 3:efbf0545:::smithi19530025-32:head [create,setxattr V\\UcZkfoVZbifk^fW\\TngQQUPi`QcehgZRpbncho (1999) in=40b,setxattr ZRreckT_dVf\\ctRT_i^dTRadaj\\ooXkpWhUg^c (4068) in=38b,setxattr ]O^Y`oiq\\set`eYfOpZ]OqpafSeUrceob[jghefP (873) in=40b,setxattr pa`OlQbdkPQV`habZkeb\\UaZhqri`qiXYgqmt_cc (1542) in=40b,setxattr OVbZnokgr`S_rPkXal\\moopUTdp[rrRYi]Oi]bpg (1297) in=40b,setxattr TRpko\\X\\jsnarS\\dkqaY\\c\\`aiVlhot[[o_O_UoR (1684) in=40b,setxattr Urbt (4068) in=4b,setxattr ^][Unhjgnmr[nWlcpqqhss__fd[WOUToconim]Pj (3084) in=40b,setxattr iU[ard`mj[Qijanah`QONRPPPPWnjV]dZmoVlOOP (1262) in=40b,setxattr O\\YgbW\\hbbUsUSYfg\\TToafkkSjn]ZRYaVQkiQme (245) in=40b,setxattr XqWdb^p[gN[RgrQQYdgqdbUS`aqdSTh_herpkti[ (173) in=40b,setxattr pWY`hiOY`QdPYT\\iOQpUiTUUXckkmUc_rkXiVOjs (699) in=40b,setxattr ncTPYl^j\\dpmpb]TiNUe`j]mohmqia\\NstbaRRgV (1927) in=40b,setxattr r^WOa`]_fNedaNN_edhqXf`eRUfUQi\\aoTWeX\\a_ (16140) in=40b,setxattr ]bbomOdb_dN_\\SYQdOr]OieQoahTqgrr\\llq\\[dg (12488) in=40b,setxattr aPcNrlTpY`tflOtRiY_UYgScbli]d^ZWQeSRsSTr (525) in=40b,setxattr XWk\\TgTZfTp`Znlj_N\\a[^cYtdocP`cYfO\\dUY`^ (490) in=40b,setxattr SabROtbgPZ[^aaQ^lVRWgQaToRknOSgk`gepYcVP (17332) in=40b,setxattr YSPXXlaUgoelQOSTk`rYfP_jsmZZqblXaeegO_cg (1329) in=40b,setxattr dphWftYOkiffV[XU^WqtifdadXcWlj (4068) in=30b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 0.92874979400000002, 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012031088000000001, 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 788 2024-09-16T02:15:20.017 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994973+0000", 2024-09-16T02:15:20.018 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5780000000000001e-06 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995089+0000", 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115374 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995112+0000", 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3201e-05 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.019 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995124+0000", 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2421e-05 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995232+0000", 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107409 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995396+0000", 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000164573 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.020 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.996898+0000", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001501549 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.003436+0000", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065382130000000002 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.006853+0000", 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003416592 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.007002+0000", 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000149178 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.022 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.024 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:20.027 INFO:tasks.rados.rados.0.smithi195.stdout:321: done (5 left) 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 44 current snap is 27 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:322: seq_num 126 ranges {245760=573440,819200=622592,1441792=622592,2064384=311296} 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:322: writing smithi19530025-44 from 245760 to 819200 tid 1 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:322: writing smithi19530025-44 from 819200 to 1441792 tid 2 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:322: writing smithi19530025-44 from 1441792 to 2064384 tid 3 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:322: writing smithi19530025-44 from 2064384 to 2375680 tid 4 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 50 current snap is 27 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:323: seq_num 127 ranges {2031616=589824,2621440=753664,3375104=475136,3850240=327680} 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:323: writing smithi19530025-50 from 2031616 to 2621440 tid 1 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:323: writing smithi19530025-50 from 2621440 to 3375104 tid 2 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:323: writing smithi19530025-50 from 3375104 to 3850240 tid 3 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:323: writing smithi19530025-50 from 3850240 to 4177920 tid 4 2024-09-16T02:15:20.028 INFO:tasks.rados.rados.0.smithi195.stdout:324: rmattr oid 34 current snap is 27 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 1 to smithi19530025-15 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:319: finishing rollback tid 0 to smithi19530025-3 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:319: finishing rollback tid 1 to smithi19530025-3 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:319: finishing rollback tid 2 to smithi19530025-3 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 3 v 69 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 2 to smithi19530025-15 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 3 to smithi19530025-15 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:318: done (7 left) 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:319: done (6 left) 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:325: rmattr oid 43 current snap is 27 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:325: done (6 left) 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 6 current snap is 27 2024-09-16T02:15:20.029 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 4 to smithi19530025-15 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 1 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 2 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 3 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:326: seq_num 128 ranges {98304=606208,704512=507904,1212416=573440,1785856=475136} 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:326: writing smithi19530025-6 from 98304 to 704512 tid 1 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:326: writing smithi19530025-6 from 704512 to 1212416 tid 2 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:326: writing smithi19530025-6 from 1212416 to 1785856 tid 3 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:326: writing smithi19530025-6 from 1785856 to 2260992 tid 4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:327: delete oid 18 current snap is 27 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 4 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 5 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 6 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: finishing write tid 7 to smithi19530025-4 2024-09-16T02:15:20.030 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 updating version 0 to 56 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 updating version 56 to 58 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 version 58 is already newer than 55 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 version 58 is already newer than 57 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 updating version 58 to 60 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: oid 4 version 60 is already newer than 59 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 4 v 60 (ObjNum 125 snap 27 seq_num 125) dirty exists 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:320: left oid 4 (ObjNum 125 snap 27 seq_num 125) 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 5 to smithi19530025-48 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: finishing write tid 6 to smithi19530025-48 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: oid 48 updating version 0 to 160 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: oid 48 updating version 160 to 163 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: oid 48 version 163 is already newer than 162 2024-09-16T02:15:20.031 INFO:tasks.rados.rados.0.smithi195.stdout:314: oid 48 updating version 163 to 164 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:314: oid 48 version 164 is already newer than 161 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 48 v 164 (ObjNum 123 snap 27 seq_num 123) dirty exists 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:314: left oid 48 (ObjNum 123 snap 27 seq_num 123) 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 5 to smithi19530025-15 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: finishing write tid 6 to smithi19530025-15 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: oid 15 updating version 0 to 93 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: oid 15 updating version 93 to 94 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: oid 15 version 94 is already newer than 91 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: oid 15 version 94 is already newer than 92 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:317: oid 15 updating version 94 to 95 2024-09-16T02:15:20.032 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 15 v 95 (ObjNum 124 snap 27 seq_num 124) dirty exists 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:317: left oid 15 (ObjNum 124 snap 27 seq_num 124) 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 1 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 2 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 3 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 4 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 5 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: finishing write tid 6 to smithi19530025-44 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: oid 44 updating version 0 to 166 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: oid 44 updating version 166 to 174 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: oid 44 version 174 is already newer than 173 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: oid 44 version 174 is already newer than 172 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: oid 44 version 174 is already newer than 171 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 174 (ObjNum 126 snap 27 seq_num 126) dirty exists 2024-09-16T02:15:20.033 INFO:tasks.rados.rados.0.smithi195.stdout:322: left oid 44 (ObjNum 126 snap 27 seq_num 126) 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 1 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 2 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 112 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 3 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 4 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 5 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: finishing write tid 6 to smithi19530025-50 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: oid 50 updating version 0 to 62 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: oid 50 version 62 is already newer than 61 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: oid 50 version 62 is already newer than 60 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: oid 50 updating version 62 to 63 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: oid 50 updating version 63 to 64 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 50 v 64 (ObjNum 127 snap 27 seq_num 127) dirty exists 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:323: left oid 50 (ObjNum 127 snap 27 seq_num 127) 2024-09-16T02:15:20.034 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 1 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 2 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 3 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 4 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 5 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: finishing write tid 6 to smithi19530025-6 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: oid 6 updating version 0 to 28 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: oid 6 updating version 28 to 29 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: oid 6 updating version 29 to 31 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: oid 6 version 31 is already newer than 27 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: oid 6 version 31 is already newer than 30 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 6 v 31 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-16T02:15:20.035 INFO:tasks.rados.rados.0.smithi195.stdout:326: left oid 6 (ObjNum 128 snap 27 seq_num 128) 2024-09-16T02:15:20.036 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:20.020+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:20.075 INFO:teuthology.orchestra.run.smithi177.stdout:reshard success 2024-09-16T02:15:20.124 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:20.122+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:20.192 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:20.192 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:20.192 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:20.192 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.206 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:20.225 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:20.220+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:20.326 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:20.322+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:20.329 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:20.329 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:20.329 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.345 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:20.427 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:20.422+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.471 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:484 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [setxattr _header (57) in=7b] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.461131275, 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62580566599999998, 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 484 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006736+0000", 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2500000000000004e-07 2024-09-16T02:15:20.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006740+0000", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6460000000000001e-06 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006749+0000", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0390000000000005e-06 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006754+0000", 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1939999999999997e-06 2024-09-16T02:15:20.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011015+0000", 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612830000000003 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011096+0000", 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0894000000000002e-05 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011534+0000", 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043832300000000001 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012404+0000", 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086940800000000001 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632372+0000", 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.619968348 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632541+0000", 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016870599999999999 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:485 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.461104902000002, 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.628869487, 2024-09-16T02:15:20.477 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 485 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006762+0000", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4499999999999997e-07 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006765+0000", 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9270000000000001e-06 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.479 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006771+0000", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3679999999999998e-06 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006777+0000", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2959999999999998e-06 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012433+0000", 2024-09-16T02:15:20.480 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056561729999999996 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012499+0000", 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6303999999999994e-05 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633369+0000", 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62087044899999999 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633608+0000", 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023843099999999999 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.635631+0000", 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020229940000000002 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.482 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 56/32 ECSubWriteReply(tid=59, last_complete=56'17, committed=1, applied=1))", 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.455138380999998, 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62052908399999995, 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012730+0000", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.694e-06 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012741+0000", 2024-09-16T02:15:20.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1343e-05 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632045+0000", 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61930391299999998 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632066+0000", 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0699999999999998e-05 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632237+0000", 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017166600000000001 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633257+0000", 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010197680000000001 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.486 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:497 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.511647461999999, 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028762988, 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 497 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956222+0000", 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6200000000000001e-06 2024-09-16T02:15:20.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956239+0000", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6458999999999999e-05 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956280+0000", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0942999999999997e-05 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956303+0000", 2024-09-16T02:15:20.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2742999999999999e-05 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956495+0000", 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000192762 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956517+0000", 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1375000000000001e-05 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.490 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962337+0000", 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058203600000000001 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962579+0000", 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024195400000000001 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.984982+0000", 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022402772000000001 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.491 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:498 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.511515938999999, 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030739767000000001, 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 498 2024-09-16T02:15:20.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956352+0000", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.75e-06 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956362+0000", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6900000000000004e-06 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956408+0000", 2024-09-16T02:15:20.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5828999999999998e-05 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956426+0000", 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8216e-05 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956684+0000", 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025823399999999998 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:20.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956702+0000", 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963967+0000", 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072649940000000003 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.966462+0000", 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024953639999999999 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.987090+0000", 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020627774000000001 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:499 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.511410978000001, 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034753205000000002, 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.497 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 499 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956457+0000", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5969999999999999e-06 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956469+0000", 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1929999999999999e-05 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956493+0000", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3849000000000002e-05 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956509+0000", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956872+0000", 2024-09-16T02:15:20.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000363424 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956891+0000", 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8913e-05 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967274+0000", 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010382466 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967524+0000", 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025058799999999999 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.991209+0000", 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023684105 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:500 3.4s0 3:23d2836d:::smithi19530025-21:head [rmxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt in=40b,rmxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ in=40b,rmxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q in=40b,rmxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd in=35b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:20.502 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.511287316999997, 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030187572999999999, 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 500 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956581+0000", 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6959999999999999e-06 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956591+0000", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.025e-05 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956614+0000", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3068999999999999e-05 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956632+0000", 2024-09-16T02:15:20.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8199999999999999e-05 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956786+0000", 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015396999999999999 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956813+0000", 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6872e-05 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963384+0000", 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065708399999999997 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963658+0000", 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027420299999999999 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965812+0000", 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021535809999999999 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.981888+0000", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016076136000000001 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986582+0000", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046937309999999996 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986767+0000", 2024-09-16T02:15:20.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018502500000000001 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 61/60 ECSubWriteReply(tid=78, last_complete=61'24, committed=1, applied=1))", 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 30.278918203, 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45998103899999998, 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.509 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188949+0000", 2024-09-16T02:15:20.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5700000000000002e-07 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188953+0000", 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5129999999999998e-06 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647595+0000", 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.458642198 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647616+0000", 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0886e-05 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647777+0000", 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016024999999999999 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.648929+0000", 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011525350000000001 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.512 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/32 ECSubWriteReply(tid=99, last_complete=71'25, committed=1, applied=1))", 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.583073084999999, 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43656151100000001, 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884795+0000", 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7230000000000001e-06 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884807+0000", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1683e-05 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320340+0000", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43553291399999999 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320354+0000", 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4953000000000001e-05 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320476+0000", 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121335 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.321355+0000", 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000878903 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:646 3.4s0 3:2f4f4374:::smithi19530025-31:head [append 0~655360] snapc 14=[13,12,d] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.512060051999999, 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031432506999999998, 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 646 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955808+0000", 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9379999999999999e-06 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956651+0000", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00084241399999999999 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956666+0000", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5089000000000001e-05 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956674+0000", 2024-09-16T02:15:20.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3680000000000002e-06 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956762+0000", 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7546000000000005e-05 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956890+0000", 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012829200000000001 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958306+0000", 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001415735 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961177+0000", 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028716689999999999 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987150+0000", 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025972928999999999 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987239+0000", 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8527000000000001e-05 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:647 3.4s0 3:2f4f4374:::smithi19530025-31:head [append 0~606208] snapc 14=[13,12,d] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 18.506658085000002, 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026701157, 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.522 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 647 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961211+0000", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5239999999999999e-06 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961712+0000", 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050093999999999996 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961728+0000", 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6379999999999999e-05 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961739+0000", 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1090999999999999e-05 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961947+0000", 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020735 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962184+0000", 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023711199999999999 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964605+0000", 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002421154 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.526 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970291+0000", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056862900000000001 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987855+0000", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017563552999999999 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987909+0000", 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4762999999999997e-05 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.527 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:708 3.bs0 3:d29cad4f:::smithi19530025-29:head [setxattr _header (57) in=7b] snapc 16=[d] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.435222505, 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029424117999999999, 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 708 2024-09-16T02:15:20.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4300000000000003e-07 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032650+0000", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1229999999999999e-06 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032654+0000", 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7310000000000001e-06 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032658+0000", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6049999999999998e-06 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.034706+0000", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020481869999999999 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.034795+0000", 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9376000000000006e-05 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035215+0000", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041973200000000003 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035984+0000", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000768598 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061988+0000", 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026003727000000001 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062068+0000", 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0395999999999994e-05 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:709 3.bs0 3:d29cad4f:::smithi19530025-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.435203059999999, 2024-09-16T02:15:20.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030374595000000001, 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 709 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:20.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032664+0000", 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.77e-07 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032666+0000", 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0049999999999999e-06 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032672+0000", 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.516e-06 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032678+0000", 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3730000000000001e-06 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035885+0000", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032076069999999999 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035962+0000", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6895000000000002e-05 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062474+0000", 2024-09-16T02:15:20.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026512285 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062564+0000", 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9628000000000003e-05 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.063038+0000", 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047390900000000001 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:719 3.4s0 3:2796292c:::smithi19530025-3:head [copy-from ver 21] snapc 16=[d] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.435180581999999, 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032989046000000001, 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 719 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.539 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:20.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4799999999999999e-07 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032689+0000", 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.368e-06 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032693+0000", 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4750000000000004e-06 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032700+0000", 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6030000000000001e-06 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032819+0000", 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119722 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033058+0000", 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000239101 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.057868+0000", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024809127 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062005+0000", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041371569999999998 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065628+0000", 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036227859999999998 2024-09-16T02:15:20.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065675+0000", 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7259000000000003e-05 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:720 3.4s0 3:2796292c:::smithi19530025-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.435152063, 2024-09-16T02:15:20.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029151192999999999, 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 720 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:20.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032715+0000", 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.75e-07 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032718+0000", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5930000000000002e-06 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032727+0000", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2369999999999992e-06 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032736+0000", 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3610000000000007e-06 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033456+0000", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072037800000000003 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033521+0000", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5143999999999998e-05 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061586+0000", 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028064787000000001 2024-09-16T02:15:20.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061711+0000", 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012470999999999999 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061865+0000", 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015450800000000001 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 79/32 ECSubWriteReply(tid=146, last_complete=79'44, committed=1, applied=1))", 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.038169+0000", 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.429697271, 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024249471000000002, 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.038169+0000", 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.038169+0000", 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.038170+0000", 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3399999999999998e-07 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.038174+0000", 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0160000000000002e-06 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061824+0000", 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023649969999999999 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061832+0000", 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2949999999999994e-06 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061890+0000", 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7822999999999998e-05 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062419+0000", 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00052863300000000003 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.553 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 79/66 ECSubWriteReply(tid=150, last_complete=79'55, committed=1, applied=1))", 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 12.400325693999999, 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.39361127800000001, 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7199999999999998e-07 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067548+0000", 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2319999999999998e-06 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.460948+0000", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.39340061199999998 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.460961+0000", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.311e-05 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.461117+0000", 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015610900000000001 2024-09-16T02:15:20.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.461152+0000", 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4542999999999998e-05 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:764 3.4s0 3:2f4f4374:::smithi19530025-31:head [setxattr _header (57) in=7b] snapc 19=[18,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3788862130000004, 2024-09-16T02:15:20.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.60102845199999999, 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 764 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088981+0000", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5499999999999998e-07 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088984+0000", 2024-09-16T02:15:20.559 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5990000000000002e-06 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088991+0000", 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.148e-06 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088997+0000", 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0459999999999996e-06 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.560 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092231+0000", 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003234823 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092325+0000", 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3385000000000001e-05 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092707+0000", 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038185900000000002 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.561 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093576+0000", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086912100000000004 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.689836+0000", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.59626043900000003 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690009+0000", 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017237700000000001 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.562 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:765 3.4s0 3:2f4f4374:::smithi19530025-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 6.3788614240000001, 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.60354616400000005, 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 765 2024-09-16T02:15:20.563 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.564 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8999999999999997e-07 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089008+0000", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5809999999999999e-06 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089016+0000", 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1689999999999992e-06 2024-09-16T02:15:20.565 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089023+0000", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4339999999999997e-06 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093437+0000", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044148850000000003 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093522+0000", 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4493999999999994e-05 2024-09-16T02:15:20.566 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690813+0000", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.59729147199999999 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690957+0000", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014378899999999999 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.692551+0000", 2024-09-16T02:15:20.567 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00159385 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 90/32 ECSubWrite(tid=127, reqid=client.4598.0:796, at_version=90'95, trim_to=0'0, roll_forward_to=90'91))", 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.437484427, 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43142370499999999, 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.568 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030383+0000", 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.235e-06 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.569 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030470+0000", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199000000000003e-05 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460364+0000", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.42989403399999998 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460379+0000", 2024-09-16T02:15:20.570 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.448e-05 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460489+0000", 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011054599999999999 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460512+0000", 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2736000000000001e-05 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:20.571 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.461736+0000", 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001223937 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.461806+0000", 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9537999999999997e-05 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:20.572 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.574 INFO:tasks.thrashosds.thrasher:running fsck to verify new sharding on osd.10 2024-09-16T02:15:20.574 DEBUG:teuthology.orchestra.run.smithi177:> sudo ceph-bluestore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/bluestore_tool.$pid.log' --log-level=10 --path /var/lib/ceph/osd/ceph-10 fsck 2024-09-16T02:15:20.575 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:20.575 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:20.570+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:20.675 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:20.668+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:20.742 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:20.742 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:20.743 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:20.743 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.758 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:20.776 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:20.773+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:20.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:20.877 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:20.874+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:20.882 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:20.882 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:20.882 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:20.882 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:20.882 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:20.897 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:314: done (7 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:317: done (6 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:320: done (5 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:322: done (4 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:323: done (3 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:324: done (2 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:326: done (1 left) 2024-09-16T02:15:20.898 INFO:tasks.rados.rados.0.smithi195.stdout:327: done (0 left) 2024-09-16T02:15:20.899 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 30 current snap is 27 2024-09-16T02:15:20.899 INFO:tasks.rados.rados.0.smithi195.stdout:328: seq_num 130 ranges {0=638976,638976=671744,1310720=802816,2113536=81920} 2024-09-16T02:15:20.902 INFO:tasks.rados.rados.0.smithi195.stdout:328: writing smithi19530025-30 from 0 to 638976 tid 1 2024-09-16T02:15:20.906 INFO:tasks.rados.rados.0.smithi195.stdout:328: writing smithi19530025-30 from 638976 to 1310720 tid 2 2024-09-16T02:15:20.911 INFO:tasks.rados.rados.0.smithi195.stdout:328: writing smithi19530025-30 from 1310720 to 2113536 tid 3 2024-09-16T02:15:20.912 INFO:tasks.rados.rados.0.smithi195.stdout:328: writing smithi19530025-30 from 2113536 to 2195456 tid 4 2024-09-16T02:15:20.912 INFO:tasks.rados.rados.0.smithi195.stdout:329: copy_from oid 25 from oid 50 current snap is 27 2024-09-16T02:15:20.912 INFO:tasks.rados.rados.0.smithi195.stdout:330: snap_create 2024-09-16T02:15:20.914 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 1 to smithi19530025-30 2024-09-16T02:15:20.920 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 2 to smithi19530025-30 2024-09-16T02:15:20.922 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 3 to smithi19530025-30 2024-09-16T02:15:20.924 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 4 to smithi19530025-30 2024-09-16T02:15:20.926 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 5 to smithi19530025-30 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: finishing write tid 6 to smithi19530025-30 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: oid 30 updating version 0 to 117 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: oid 30 version 117 is already newer than 113 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: oid 30 version 117 is already newer than 116 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: oid 30 version 117 is already newer than 115 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:328: oid 30 version 117 is already newer than 114 2024-09-16T02:15:20.929 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 117 (ObjNum 130 snap 27 seq_num 130) dirty exists 2024-09-16T02:15:20.930 INFO:tasks.rados.rados.0.smithi195.stdout:328: left oid 30 (ObjNum 130 snap 27 seq_num 130) 2024-09-16T02:15:20.955 INFO:tasks.rados.rados.0.smithi195.stdout:329: finishing copy_from racing read to smithi19530025-25 2024-09-16T02:15:20.956 INFO:tasks.rados.rados.0.smithi195.stdout:329: finishing copy_from to smithi19530025-25 2024-09-16T02:15:20.956 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 25 v 64 (ObjNum 127 snap 27 seq_num 127) dirty exists 2024-09-16T02:15:20.957 INFO:tasks.rados.rados.0.smithi195.stdout:328: done (2 left) 2024-09-16T02:15:20.957 INFO:tasks.rados.rados.0.smithi195.stdout:329: done (1 left) 2024-09-16T02:15:20.958 INFO:tasks.rados.rados.0.smithi195.stdout:330: done (0 left) 2024-09-16T02:15:20.958 INFO:tasks.rados.rados.0.smithi195.stdout:331: delete oid 36 current snap is 28 2024-09-16T02:15:20.970 INFO:tasks.rados.rados.0.smithi195.stdout:331: done (0 left) 2024-09-16T02:15:20.970 INFO:tasks.rados.rados.0.smithi195.stdout:332: snap_create 2024-09-16T02:15:20.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:15:20.977 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:20.974+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:21.010 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:21.010 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:33 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.727972141999999, 2024-09-16T02:15:21.021 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024514080000000001, 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 33 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.022 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291064+0000", 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.023 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291067+0000", 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7199999999999998e-06 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291074+0000", 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4680000000000003e-06 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291080+0000", 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.964e-06 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.024 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055391950000000002 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0838000000000001e-05 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313405+0000", 2024-09-16T02:15:21.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016704404999999999 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313545+0000", 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014004999999999999 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315578+0000", 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020328199999999999 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.026 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:39 3.1s0 3:81c6c0a8:::smithi19530025-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.727602277000003, 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024374809000000001, 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.027 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 39 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:21.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291434+0000", 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291440+0000", 2024-09-16T02:15:21.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4960000000000004e-06 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291456+0000", 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025e-05 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291468+0000", 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299421+0000", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079526309999999999 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299510+0000", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9506000000000007e-05 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314495+0000", 2024-09-16T02:15:21.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014984548 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314572+0000", 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6854999999999995e-05 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315808+0000", 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012362639999999999 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.032 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:40 3.1s0 3:830a71a8:::smithi19530025-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.727550684999997, 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024497693000000001, 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.033 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 40 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291486+0000", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8999999999999995e-07 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291773+0000", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028741999999999998 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291795+0000", 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.181e-05 2024-09-16T02:15:21.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291807+0000", 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1687e-05 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299547+0000", 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077406699999999998 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299647+0000", 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9978999999999999e-05 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.300445+0000", 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079766400000000001 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302061+0000", 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016158539999999999 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315913+0000", 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013852573999999999 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315983+0000", 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9145000000000003e-05 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:41 3.1s0 3:830a71a8:::smithi19530025-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.727203351, 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024477925000000001, 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.039 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291833+0000", 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291839+0000", 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291855+0000", 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291865+0000", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0037999999999999e-05 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301194+0000", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093284930000000002 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301281+0000", 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6972999999999996e-05 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301794+0000", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051343099999999996 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.304892+0000", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030976950000000001 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316223+0000", 2024-09-16T02:15:21.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011331071999999999 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316310+0000", 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.742e-05 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:42 3.1s0 3:830a71a8:::smithi19530025-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:21.044 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.727157159000001, 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031481841000000003, 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:21.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291880+0000", 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2139999999999999e-06 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291883+0000", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291902+0000", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8881000000000001e-05 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291912+0000", 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9680000000000001e-06 2024-09-16T02:15:21.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302284+0000", 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371521999999999 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302334+0000", 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0047999999999999e-05 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318985+0000", 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016650888999999999 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.319121+0000", 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136336 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323360+0000", 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042391670000000003 2024-09-16T02:15:21.049 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:247 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (56) in=7b] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.588474347000002, 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30233062700000002, 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 247 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:21.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430562+0000", 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1230000000000001e-06 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430568+0000", 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.519e-06 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430582+0000", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.366e-05 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430592+0000", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0482999999999999e-05 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433637+0000", 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003044705 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433734+0000", 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6916000000000005e-05 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434127+0000", 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039313800000000002 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435690+0000", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015630640000000001 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732747+0000", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29705759700000001 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732892+0000", 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014442200000000001 2024-09-16T02:15:21.055 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:248 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 47.588399903999999, 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.304991454, 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 248 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:21.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430637+0000", 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3099999999999996e-07 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430641+0000", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430654+0000", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638999999999999e-05 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430662+0000", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7370000000000006e-06 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434972+0000", 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043098019999999997 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435055+0000", 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3625e-05 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.733794+0000", 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.298738691 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.734027+0000", 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023328199999999999 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.735627+0000", 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015998379999999999 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:250 3.1s0 3:89ce0a26:::smithi19530025-11:head [copy-from ver 2] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.673975151999997, 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046806635999999999, 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 250 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.062 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345064+0000", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.484e-06 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345079+0000", 2024-09-16T02:15:21.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4939999999999999e-05 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345114+0000", 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5258e-05 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345131+0000", 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7356000000000001e-05 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345312+0000", 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018022200000000001 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345332+0000", 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0509e-05 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011526471 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357011+0000", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015221500000000001 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375994+0000", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018982974999999999 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380796+0000", 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048023010000000001 2024-09-16T02:15:21.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391742+0000", 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010946141 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391867+0000", 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124764 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:251 3.1s0 3:89ce0a26:::smithi19530025-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.673857546000001, 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034174018, 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 251 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.069 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345179+0000", 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1799999999999999e-06 2024-09-16T02:15:21.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345186+0000", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5259999999999999e-06 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345202+0000", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6453999999999999e-05 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345213+0000", 2024-09-16T02:15:21.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1147e-05 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345556+0000", 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000342219 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345581+0000", 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5876000000000001e-05 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012465654 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358190+0000", 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014254599999999999 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379056+0000", 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020865887 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379212+0000", 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156675 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379352+0000", 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013985399999999999 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.074 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 45/32 ECSubWriteReply(tid=55, last_complete=45'49, committed=1, applied=1))", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.626038530000002, 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.456257309, 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392998+0000", 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2780000000000001e-06 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.393006+0000", 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1310000000000006e-06 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847561+0000", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45455489999999998 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847576+0000", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.453e-05 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847695+0000", 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118799 2024-09-16T02:15:21.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.849254+0000", 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015596710000000001 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:260 3.1s0 3:830a71a8:::smithi19530025-14:head [copy-from ver 2] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.661926964000003, 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02650228, 2024-09-16T02:15:21.078 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 260 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357110+0000", 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.243e-06 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357117+0000", 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9e-06 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357132+0000", 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5156e-05 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357141+0000", 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357434+0000", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029301799999999998 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357457+0000", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2821999999999999e-05 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365347+0000", 2024-09-16T02:15:21.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007890088 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365650+0000", 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030261099999999999 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.379802+0000", 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014151762 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381743+0000", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001941531 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383415+0000", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016721349999999999 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383611+0000", 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019561399999999999 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.084 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:261 3.1s0 3:830a71a8:::smithi19530025-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.661846136999998, 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024496032000000001, 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.085 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 261 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357191+0000", 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3799999999999999e-06 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357198+0000", 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1740000000000003e-06 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.087 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357220+0000", 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2322000000000001e-05 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357238+0000", 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7377e-05 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357578+0000", 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034031900000000001 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357600+0000", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1543e-05 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366106+0000", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085068360000000003 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366241+0000", 2024-09-16T02:15:21.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013419899999999999 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381121+0000", 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014880769 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381357+0000", 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023564300000000001 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381685+0000", 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032846999999999998 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:501 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 284996~1533628,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.062173129000001, 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023246494999999999, 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.091 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 501 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956862+0000", 2024-09-16T02:15:21.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956866+0000", 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6500000000000002e-06 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956875+0000", 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5469999999999997e-06 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956912+0000", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6862e-05 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956935+0000", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3986e-05 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957148+0000", 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021242200000000001 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957166+0000", 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7604999999999998e-05 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969486+0000", 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012320806 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969825+0000", 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00033867100000000002 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.980109+0000", 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010283946 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:502 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1658552~160072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.062040971999998, 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025237151999999999, 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 502 2024-09-16T02:15:21.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956996+0000", 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.28e-06 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957005+0000", 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8459999999999997e-06 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957029+0000", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4053000000000002e-05 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957045+0000", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6293000000000001e-05 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957336+0000", 2024-09-16T02:15:21.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029122600000000001 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957354+0000", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7543000000000001e-05 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974088+0000", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016734532999999999 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974356+0000", 2024-09-16T02:15:21.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026803599999999998 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.982232+0000", 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078753420000000005 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:503 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1451430~367194,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.061915761999998, 2024-09-16T02:15:21.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.32090356199999998, 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 503 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:21.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957122+0000", 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3470000000000001e-06 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957136+0000", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4079e-05 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957168+0000", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2159000000000002e-05 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957189+0000", 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0367e-05 2024-09-16T02:15:21.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957429+0000", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000240561 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957446+0000", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6472000000000001e-05 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976521+0000", 2024-09-16T02:15:21.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019075392 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976775+0000", 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025338100000000001 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278023+0000", 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30124880399999998 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpReadReply(3.1s0 58/32 ECSubReadReply(tid=123, attrs_read=0))", 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.040561412999999, 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29957913899999999, 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978475+0000", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2999999999999999e-07 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978563+0000", 2024-09-16T02:15:21.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8211e-05 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277175+0000", 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29861169199999998 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277197+0000", 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1566999999999999e-05 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277332+0000", 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013569999999999999 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278053+0000", 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072103900000000003 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es4 86/83 ECSubWrite(tid=95, reqid=client.4598.0:767, at_version=0'0, trim_to=0'0, roll_forward_to=86'56))", 2024-09-16T02:15:21.111 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.918793859, 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83795019199999998, 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100257+0000", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5393e-05 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100278+0000", 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0883e-05 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936352+0000", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83607375399999995 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936368+0000", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6503000000000001e-05 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936508+0000", 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013982799999999999 2024-09-16T02:15:21.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936525+0000", 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7042999999999999e-05 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.938115+0000", 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001589795 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.938192+0000", 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6993e-05 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:777 3.1s0 3:8203d376:::smithi19530025-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e87)", 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.0399485449999997, 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023317917000000001, 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.116 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 777 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979089+0000", 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1299999999999999e-06 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979098+0000", 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3529999999999995e-06 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979121+0000", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3329e-05 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979143+0000", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1656000000000001e-05 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979310+0000", 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167744 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979325+0000", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4363e-05 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985273+0000", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059480979999999998 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985511+0000", 2024-09-16T02:15:21.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000238748 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.001858+0000", 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016346612999999999 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.002120+0000", 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026181300000000001 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.002405+0000", 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028507000000000001 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:786 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.0274328260000001, 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.113603414, 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.122 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 786 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:21.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991604+0000", 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4839999999999999e-06 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991627+0000", 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2489000000000001e-05 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991645+0000", 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7861999999999999e-05 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991656+0000", 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0994e-05 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995156+0000", 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035008109999999999 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995476+0000", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000319609 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.996848+0000", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013722630000000001 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.000177+0000", 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033282540000000001 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.105020+0000", 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104843271 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.105206+0000", 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018637699999999999 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.127 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:787 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 2.0273610450000001, 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116105888, 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 787 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.128 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991676+0000", 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.04e-06 2024-09-16T02:15:21.129 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991680+0000", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2799999999999997e-06 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991692+0000", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1663e-05 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991700+0000", 2024-09-16T02:15:21.130 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2160000000000006e-06 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.997760+0000", 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060602829999999996 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.998015+0000", 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025511099999999998 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.131 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.106099+0000", 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10808414700000001 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.106254+0000", 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015439100000000001 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.107780+0000", 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015267569999999999 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.132 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.133 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.133 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.133 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:21.134 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:21.134 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:21.135 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:21.135 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:21.135 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:21.136 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:21.136 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:21.136 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:21.137 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:21.137 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:21.134+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:21.137 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:21.137 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:21.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:21.165 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:21.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:21.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:21.217 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:21.218 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:21.235 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:21.234+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:21.246 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:21.246 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:21.247 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:21.276 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:21.276 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:21.303 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:21.303 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:21.311 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:21.311 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:21.311 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:21.311 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:21.325 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:21.333 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:21.333 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:21.336 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:21.333+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:21.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:21.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:21.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:21.362+0000 7f4461d9b640 0 -- 172.21.15.3:0/2399269438 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f4464224810 msgr2=0x7f4464226c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:15:21.437 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:21.432+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:21.444 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:21.444 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:21.444 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:21.444 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:21.444 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:21.459 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:21.538 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:21.538+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:21.583 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.300921148, 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.584 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.585 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.586 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:15:21.587 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.588 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.589 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.300728372999998, 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.590 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:21.591 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.592 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:15:21.593 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:15:21.594 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.595 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.300670330000003, 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:15:21.596 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.597 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:15:21.598 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:15:21.599 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.600 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.601 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.271513400000003, 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.602 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.603 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.604 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:15:21.605 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.606 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.271460341000001, 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.607 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.608 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:15:21.609 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.610 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.611 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:15:21.612 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.236278878, 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.613 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.614 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.615 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:15:21.616 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.223066180000004, 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.617 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.618 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:15:21.619 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.620 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:15:21.621 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.622 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 46.222956943, 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.624 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:15:21.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:15:21.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.629 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:21.630 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 40.673692762000002, 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:15:21.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:15:21.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:15:21.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:21.636 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 40.673577035000001, 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:15:21.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:15:21.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:15:21.641 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 40.652042252000001, 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:15:21.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.645 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 73/66 ECSubWrite(tid=66, reqid=osd.8.0:66, at_version=73'42, trim_to=0'0, roll_forward_to=72'38))", 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.625210206999999, 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018190284000000001, 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:21.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954947+0000", 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5069999999999999e-06 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954992+0000", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4162000000000002e-05 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955233+0000", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000241566 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955251+0000", 2024-09-16T02:15:21.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7448999999999998e-05 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956483+0000", 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012321509999999999 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956507+0000", 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973035+0000", 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016528547000000001 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973136+0000", 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010087200000000001 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.650 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 73/32 ECSubWrite(tid=107, reqid=osd.5.0:107, at_version=73'75, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.621501666, 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016895095999999998, 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958656+0000", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5939999999999999e-06 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958707+0000", 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1008999999999999e-05 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958930+0000", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022282899999999999 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958952+0000", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2146000000000001e-05 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959126+0000", 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173807 2024-09-16T02:15:21.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959156+0000", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0022000000000002e-05 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975497+0000", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016340924999999999 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975550+0000", 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2763999999999998e-05 2024-09-16T02:15:21.654 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 19.616055412000001, 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014562088000000001, 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964103+0000", 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.26e-06 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.656 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964341+0000", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023783300000000001 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964452+0000", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110974 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964476+0000", 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.393e-05 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964624+0000", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014846400000000001 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964652+0000", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7389000000000002e-05 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978619+0000", 2024-09-16T02:15:21.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013967228 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978663+0000", 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.401e-05 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 13.514325381999999, 2024-09-16T02:15:21.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79668697799999999, 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065832+0000", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4780000000000001e-06 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065862+0000", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9570999999999999e-05 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860567+0000", 2024-09-16T02:15:21.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79470507099999999 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860588+0000", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0665e-05 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860756+0000", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016797700000000001 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860787+0000", 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1492000000000003e-05 2024-09-16T02:15:21.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862431+0000", 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016435880000000001 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862518+0000", 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7136000000000001e-05 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=103, reqid=client.4598.0:770, at_version=87'86, trim_to=0'0, roll_forward_to=79'84))", 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.6214524560000001, 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017445961999999999, 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:21.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958706+0000", 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1660000000000001e-06 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.959284+0000", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057834499999999997 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972561+0000", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01327654 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972585+0000", 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4221999999999999e-05 2024-09-16T02:15:21.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972741+0000", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156248 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972764+0000", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3136e-05 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976076+0000", 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003311858 2024-09-16T02:15:21.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976150+0000", 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3447000000000003e-05 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=107, reqid=client.4598.0:774, at_version=87'90, trim_to=0'0, roll_forward_to=87'86))", 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:21.668 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.5949539020000003, 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19572739, 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:21.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985204+0000", 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2630000000000002e-06 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985253+0000", 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8288000000000003e-05 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179555+0000", 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.194302157 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179570+0000", 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5322999999999999e-05 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179746+0000", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175391 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179762+0000", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6388999999999998e-05 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180848+0000", 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010859349999999999 2024-09-16T02:15:21.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180930+0000", 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1644000000000006e-05 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:831 3.6s0 3:63499ffe:::smithi19530025-18:head [delete] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91)", 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.574317626, 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89186369399999998, 2024-09-16T02:15:21.673 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 831 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0999999999999997e-07 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005846+0000", 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1690000000000003e-06 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005911+0000", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4146000000000006e-05 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005928+0000", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7541999999999999e-05 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008320+0000", 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002391476 2024-09-16T02:15:21.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008367+0000", 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.757e-05 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008731+0000", 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036390000000000001 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:21.677 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.009626+0000", 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00089470299999999995 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897500+0000", 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88787382500000001 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897702+0000", 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020255300000000001 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.570105541, 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066496243999999996, 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010052+0000", 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0899999999999999e-06 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010064+0000", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2726e-05 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076273+0000", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066208820000000002 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076290+0000", 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6654e-05 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076469+0000", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178717 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076547+0000", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8237000000000003e-05 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:21.682 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 1.5692258830000001, 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88785698300000004, 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:21.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010932+0000", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.48e-06 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010939+0000", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897090+0000", 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88615089199999997 2024-09-16T02:15:21.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897108+0000", 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.77e-05 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897272+0000", 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016364099999999999 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:21.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.898787+0000", 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015155539999999999 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:21.686 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:21.687 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:21.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:21.689 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:21.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:21.690 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:21.685+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:21.789 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:21.784+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:21.865 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:21.865 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:21.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:21.889 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:21.886+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:21.958 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:15:21.963 INFO:tasks.rados.rados.0.smithi195.stdout:332: done (0 left) 2024-09-16T02:15:21.963 INFO:tasks.rados.rados.0.smithi195.stdout:333: setattr oid 34 current snap is 29 2024-09-16T02:15:21.968 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 11 current snap is 29 2024-09-16T02:15:21.968 INFO:tasks.rados.rados.0.smithi195.stdout:334: seq_num 132 ranges {1818624=671744,2490368=573440,3063808=753664,3817472=229376} 2024-09-16T02:15:21.990 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:21.990+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:21.998 INFO:tasks.rados.rados.0.smithi195.stdout:334: writing smithi19530025-11 from 1818624 to 2490368 tid 1 2024-09-16T02:15:22.004 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.004 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.006 INFO:tasks.rados.rados.0.smithi195.stdout:334: writing smithi19530025-11 from 2490368 to 3063808 tid 2 2024-09-16T02:15:22.016 INFO:tasks.rados.rados.0.smithi195.stdout:334: writing smithi19530025-11 from 3063808 to 3817472 tid 3 2024-09-16T02:15:22.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:22.019 INFO:tasks.rados.rados.0.smithi195.stdout:334: writing smithi19530025-11 from 3817472 to 4046848 tid 4 2024-09-16T02:15:22.019 INFO:tasks.rados.rados.0.smithi195.stdout:335: delete oid 41 current snap is 29 2024-09-16T02:15:22.019 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 34 v 121 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-16T02:15:22.020 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 1 to smithi19530025-11 2024-09-16T02:15:22.020 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 2 to smithi19530025-11 2024-09-16T02:15:22.029 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 3 to smithi19530025-11 2024-09-16T02:15:22.033 INFO:teuthology.orchestra.run.smithi177.stdout:fsck success 2024-09-16T02:15:22.038 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 4 to smithi19530025-11 2024-09-16T02:15:22.038 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 5 to smithi19530025-11 2024-09-16T02:15:22.041 INFO:tasks.rados.rados.0.smithi195.stdout:334: finishing write tid 6 to smithi19530025-11 2024-09-16T02:15:22.041 INFO:tasks.rados.rados.0.smithi195.stdout:334: oid 11 updating version 0 to 182 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: oid 11 version 182 is already newer than 178 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: oid 11 version 182 is already newer than 180 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: oid 11 version 182 is already newer than 181 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: oid 11 version 182 is already newer than 179 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 11 v 182 (ObjNum 132 snap 29 seq_num 132) dirty exists 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: left oid 11 (ObjNum 132 snap 29 seq_num 132) 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:333: done (2 left) 2024-09-16T02:15:22.042 INFO:tasks.rados.rados.0.smithi195.stdout:334: done (1 left) 2024-09-16T02:15:22.043 INFO:tasks.rados.rados.0.smithi195.stdout:335: done (0 left) 2024-09-16T02:15:22.043 INFO:tasks.rados.rados.0.smithi195.stdout:336: snap_create 2024-09-16T02:15:22.046 INFO:tasks.thrashosds.thrasher:resharding successfully completed 2024-09-16T02:15:22.047 INFO:tasks.thrashosds.thrasher:Reviving osd 10 2024-09-16T02:15:22.047 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-16T02:15:22.047 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-16T02:15:22.091 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:22.084+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:22.091 INFO:tasks.ceph.osd.10:Started 2024-09-16T02:15:22.091 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:22.188 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.188 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.192 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:22.190+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:22.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:22.212 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-09-16T02:15:22.212 INFO:teuthology.orchestra.run.smithi177.stderr:0 2024-09-16T02:15:22.212 INFO:teuthology.orchestra.run.smithi177.stderr:1 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:2 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:abort 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:assert 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:config diff 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:config diff get 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:config get 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:config help [] 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:config set ... 2024-09-16T02:15:22.213 INFO:teuthology.orchestra.run.smithi177.stderr:admin_socket: invalid command 2024-09-16T02:15:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:15:22.215 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-16T02:15:22.293 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:22.288+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:22.330 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.330 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:22.394 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:22.394+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:22.470 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.470 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:22.494 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:22.493+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:22.595 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:22.593+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:22.656 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.657 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:22.718 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:22.718+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:22.790 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.791 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.797 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:22.794+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:22.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:22.898 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:22.896+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:22.972 INFO:tasks.rados.rados.0.smithi195.stdout:336: done (0 left) 2024-09-16T02:15:22.972 INFO:tasks.rados.rados.0.smithi195.stdout:337: snap_create 2024-09-16T02:15:22.976 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:22.976 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:22.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:22.999 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:22.994+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:23.100 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:23.098+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:23.118 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.118 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:23.201 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:23.198+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:23.223 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:23.221+0000 7f2af03cb7c0 -1 Falling back to public interface 2024-09-16T02:15:23.259 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.259 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:23.302 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:23.296+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:23.403 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:23.400+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:23.442 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.442 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:23.504 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:23.502+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:23.575 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.575 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:23.606 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:23.605+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:23.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:23.707 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:23.705+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:23.724 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.724 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:23.808 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:23.806+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:23.860 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:23.861 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:23.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:23.865 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:23.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:23.894 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:23.894 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:23.909 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:23.906+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:23.924 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:23.957 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:23.976 INFO:tasks.rados.rados.0.smithi195.stdout:337: done (0 left) 2024-09-16T02:15:23.976 INFO:tasks.rados.rados.0.smithi195.stdout:338: copy_from oid 30 from oid 29 current snap is 31 2024-09-16T02:15:23.977 INFO:tasks.rados.rados.0.smithi195.stdout:339: setattr oid 31 current snap is 31 2024-09-16T02:15:23.980 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 47 current snap is 31 2024-09-16T02:15:23.980 INFO:tasks.rados.rados.0.smithi195.stdout:340: seq_num 134 ranges {1507328=704512,2211840=475136,2686976=573440,3260416=507904} 2024-09-16T02:15:23.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:15:23.990 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:23.990 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:24.008 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 45/32 ECSubWrite(tid=17, reqid=osd.8.0:17, at_version=45'13, trim_to=0'0, roll_forward_to=36'11))", 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.640456788999998, 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056123190000000002, 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362969+0000", 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362970+0000", 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.44e-07 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363028+0000", 2024-09-16T02:15:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7784000000000001e-05 2024-09-16T02:15:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364099+0000", 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00107157 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364121+0000", 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2206000000000001e-05 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364291+0000", 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017008400000000001 2024-09-16T02:15:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364317+0000", 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5596e-05 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368520+0000", 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042029670000000002 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368582+0000", 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1568000000000004e-05 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 45/32 ECSubWrite(tid=29, reqid=osd.7.0:28, at_version=0'0, trim_to=0'0, roll_forward_to=45'10))", 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 49.640297363999998, 2024-09-16T02:15:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053661100000000003, 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363129+0000", 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363130+0000", 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5600000000000004e-07 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363151+0000", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1073e-05 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364189+0000", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010384350000000001 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364205+0000", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5732000000000001e-05 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364314+0000", 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010873400000000001 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364328+0000", 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4525000000000001e-05 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368397+0000", 2024-09-16T02:15:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040685349999999999 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368495+0000", 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8120000000000002e-05 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=22, reqid=client.4598.0:404, at_version=52'7, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.035796578000003, 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013916968, 2024-09-16T02:15:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967631+0000", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967931+0000", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030039800000000001 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979030+0000", 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011099121999999999 2024-09-16T02:15:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979051+0000", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0577e-05 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979192+0000", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140526 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979207+0000", 2024-09-16T02:15:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981459+0000", 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022516110000000002 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981546+0000", 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7508000000000003e-05 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=25, reqid=client.4598.0:407, at_version=52'10, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 43.019213592, 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34897630299999999, 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984214+0000", 2024-09-16T02:15:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.384e-06 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984244+0000", 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0549000000000002e-05 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331741+0000", 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34749696499999999 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331756+0000", 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4873e-05 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331921+0000", 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016467900000000001 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331952+0000", 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0895000000000002e-05 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333108+0000", 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001156633 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333189+0000", 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0325000000000001e-05 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.047674291, 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060351959999999996, 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955857+0000", 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010480999999999999 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955998+0000", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014078300000000001 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956026+0000", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8285000000000001e-05 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956200+0000", 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174165 2024-09-16T02:15:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956232+0000", 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1047000000000003e-05 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961698+0000", 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054662360000000002 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961787+0000", 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9234000000000007e-05 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.043801256999998, 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060372860000000002, 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959627+0000", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9099999999999999e-06 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959685+0000", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8099000000000001e-05 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959791+0000", 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106409 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959814+0000", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3224999999999999e-05 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960393+0000", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057815700000000004 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960417+0000", 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4808e-05 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965585+0000", 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051671030000000001 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965662+0000", 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7575000000000002e-05 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/32 ECSubWrite(tid=277, reqid=osd.13.0:277, at_version=74'88, trim_to=0'0, roll_forward_to=73'84))", 2024-09-16T02:15:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.0427672, 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057458700000000001, 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960660+0000", 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0400000000000005e-07 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960691+0000", 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1438e-05 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960918+0000", 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227304 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960949+0000", 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0141000000000002e-05 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962083+0000", 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011340129999999999 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962117+0000", 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3921999999999998e-05 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966316+0000", 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041993250000000003 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966405+0000", 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8923000000000002e-05 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 74/32 ECSubWrite(tid=114, reqid=osd.7.0:114, at_version=74'29, trim_to=0'0, roll_forward_to=73'27))", 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.037907271000002, 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085142380000000004, 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965520+0000", 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4e-07 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965569+0000", 2024-09-16T02:15:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9175999999999998e-05 2024-09-16T02:15:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965669+0000", 2024-09-16T02:15:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9946999999999997e-05 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965692+0000", 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3422e-05 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965858+0000", 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016617800000000001 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965887+0000", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8781000000000001e-05 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973464+0000", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075773079999999996 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974033+0000", 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056858600000000005 2024-09-16T02:15:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=112, reqid=client.4598.0:655, at_version=74'82, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.034747982999999, 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062268389999999996, 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968680+0000", 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.903e-06 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968727+0000", 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7139999999999999e-05 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968819+0000", 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1711000000000003e-05 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968841+0000", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2314e-05 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000169872 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969038+0000", 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6755e-05 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974822+0000", 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005783697 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974905+0000", 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3447000000000003e-05 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 74/66 ECSubWrite(tid=68, reqid=client.4598.0:653, at_version=74'44, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.030307385, 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069659270000000002, 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973120+0000", 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7399999999999991e-07 2024-09-16T02:15:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973262+0000", 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000142483 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973686+0000", 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042359299999999999 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973721+0000", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5491000000000003e-05 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973886+0000", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164372 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974322+0000", 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000436556 2024-09-16T02:15:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.979991+0000", 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056687619999999999 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980085+0000", 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3696000000000006e-05 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 75/66 ECSubWrite(tid=119, reqid=client.4598.0:660, at_version=75'42, trim_to=0'0, roll_forward_to=74'40))", 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.041738233, 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053709489999999999, 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961688+0000", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9299999999999999e-07 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961729+0000", 2024-09-16T02:15:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0330999999999997e-05 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961826+0000", 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7479999999999995e-05 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.961855+0000", 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8792000000000001e-05 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.962022+0000", 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016759199999999999 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.962055+0000", 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2801e-05 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.966964+0000", 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00490859 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:03.967059+0000", 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4969999999999994e-05 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 79/32 ECSubWrite(tid=138, reqid=osd.7.0:138, at_version=79'38, trim_to=0'0, roll_forward_to=78'36))", 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.013170971000001, 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079362210000000002, 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990256+0000", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990299+0000", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2870999999999997e-05 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990371+0000", 2024-09-16T02:15:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2208999999999999e-05 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990390+0000", 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8802000000000001e-05 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990544+0000", 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154661 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990576+0000", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1655000000000001e-05 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998130+0000", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075539659999999996 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998191+0000", 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1407000000000003e-05 2024-09-16T02:15:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.011732631000001, 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068584240000000001, 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991694+0000", 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-16T02:15:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991735+0000", 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1068000000000001e-05 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991880+0000", 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014438400000000001 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991902+0000", 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2616000000000002e-05 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992060+0000", 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015784600000000001 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992086+0000", 2024-09-16T02:15:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6239000000000001e-05 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998446+0000", 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063592730000000004 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998552+0000", 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106121 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 85/83 ECSubWrite(tid=170, reqid=osd.7.0:170, at_version=85'54, trim_to=0'0, roll_forward_to=84'50))", 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.069739+0000", 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.933687205, 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054733530000000002, 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069739+0000", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069739+0000", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069741+0000", 2024-09-16T02:15:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.719e-06 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069792+0000", 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1107999999999999e-05 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069879+0000", 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7519e-05 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.069902+0000", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3252e-05 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.071227+0000", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001324588 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.071255+0000", 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7792000000000001e-05 2024-09-16T02:15:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075158+0000", 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039031119999999998 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075212+0000", 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4262999999999999e-05 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 85/66 ECSubWrite(tid=174, reqid=client.4598.0:762, at_version=85'59, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.091085+0000", 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 9.9123413469999999, 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056223330000000002, 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091085+0000", 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091085+0000", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091086+0000", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0999999999999997e-07 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091553+0000", 2024-09-16T02:15:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00046739999999999998 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091792+0000", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023939799999999999 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.091844+0000", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1421000000000001e-05 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.092058+0000", 2024-09-16T02:15:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021457599999999999 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.092130+0000", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1383000000000004e-05 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.096620+0000", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044902759999999996 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.096707+0000", 2024-09-16T02:15:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6969e-05 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 90/66 ECSubWrite(tid=179, reqid=client.4598.0:789, at_version=90'63, trim_to=0'0, roll_forward_to=85'61))", 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.9976690689999996, 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077930029999999997, 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005758+0000", 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:15:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005822+0000", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4696999999999994e-05 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005935+0000", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011267799999999999 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005959+0000", 2024-09-16T02:15:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4111999999999999e-05 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006140+0000", 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018136899999999999 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006171+0000", 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0889000000000002e-05 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.013473+0000", 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073015670000000001 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.013550+0000", 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7096999999999997e-05 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 91/66 ECSubWrite(tid=187, reqid=osd.7.0:187, at_version=91'71, trim_to=0'0, roll_forward_to=90'69))", 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0534879610000001, 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070490179999999998, 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949939+0000", 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.23e-07 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950059+0000", 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119615 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950166+0000", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107125 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950187+0000", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950364+0000", 2024-09-16T02:15:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017645900000000001 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950387+0000", 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3547999999999999e-05 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956892+0000", 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065050230000000004 2024-09-16T02:15:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956987+0000", 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4739000000000002e-05 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 91/66 ECSubWrite(tid=188, reqid=osd.7.0:188, at_version=91'73, trim_to=0'0, roll_forward_to=90'69))", 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.052092955, 2024-09-16T02:15:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074890160000000002, 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951335+0000", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.623e-06 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951384+0000", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9150999999999997e-05 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951468+0000", 2024-09-16T02:15:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4196999999999994e-05 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951490+0000", 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2166000000000001e-05 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951850+0000", 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035987699999999998 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951873+0000", 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3329e-05 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.958741+0000", 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068677360000000002 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.958822+0000", 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0937000000000001e-05 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 91/83 ECSubWrite(tid=100, reqid=client.4598.0:820, at_version=91'62, trim_to=0'0, roll_forward_to=91'60))", 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.995333+0000", 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0080928419999999, 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053997890000000003, 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995333+0000", 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995333+0000", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995335+0000", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9099999999999999e-06 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995502+0000", 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016672900000000001 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995619+0000", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117475 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995642+0000", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3016000000000001e-05 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995825+0000", 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018217900000000001 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.995841+0000", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6456e-05 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.000649+0000", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004808115 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.000733+0000", 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3908999999999999e-05 2024-09-16T02:15:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s2 91/32 ECSubWrite(tid=47, reqid=client.4598.0:826, at_version=91'28, trim_to=0'0, roll_forward_to=84'25))", 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:20.001135+0000", 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.0022910859999996, 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055422830000000003, 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001135+0000", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001135+0000", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001137+0000", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.88e-06 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.003469+0000", 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023317559999999999 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.003578+0000", 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010922899999999999 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.003617+0000", 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8897999999999999e-05 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.003737+0000", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119954 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.003753+0000", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.63e-05 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.006612+0000", 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028593809999999998 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.006677+0000", 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4885e-05 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.105 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:24.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:24.107 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:24.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:24.108 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:24.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:24.109 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:24.100+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:24.109 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:24.110 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:340: writing smithi19530025-47 from 1507328 to 2211840 tid 1 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:340: writing smithi19530025-47 from 2211840 to 2686976 tid 2 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:340: writing smithi19530025-47 from 2686976 to 3260416 tid 3 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:340: writing smithi19530025-47 from 3260416 to 3768320 tid 4 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:341: read oid 36 snap -1 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:341: expect deleted 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:342: snap_create 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 75 (ObjNum 120 snap 23 seq_num 120) dirty exists 2024-09-16T02:15:24.110 INFO:tasks.rados.rados.0.smithi195.stdout:338: finishing copy_from racing read to smithi19530025-30 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 1 to smithi19530025-47 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 2 to smithi19530025-47 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:338: finishing copy_from to smithi19530025-30 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 48 (ObjNum 117 snap 21 seq_num 117) dirty exists 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 3 to smithi19530025-47 2024-09-16T02:15:24.111 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 4 to smithi19530025-47 2024-09-16T02:15:24.111 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:24.112 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:24.124 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:24.124 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:24.125 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:24.151 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:24.151 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:24.173 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:24.173 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:24.196 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:24.196 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:24.207 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:24.205+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:24.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:24.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:24.222+0000 7fed227fc640 0 -- 172.21.15.3:0/4002272667 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7fed24222820 msgr2=0x7fed24224c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:24.278 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:24.279 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:24.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:24.308 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:24.306+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:24.409 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:24.406+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:24.416 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:24.416 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:24.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:24.458 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:24.459 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:24.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:24.510 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:24.504+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:192 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~557056] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.237688831, 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015517632999999999, 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 192 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314809+0000", 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.314812+0000", 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0029999999999999e-06 2024-09-16T02:15:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315788+0000", 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097551599999999995 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315825+0000", 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6742000000000002e-05 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.315847+0000", 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2396e-05 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316018+0000", 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000170729 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.316179+0000", 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016178000000000001 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.318923+0000", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002743381 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.326096+0000", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071732469999999998 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330202+0000", 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004106391 2024-09-16T02:15:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.330327+0000", 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012444800000000001 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:195 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 57.230415321000002, 2024-09-16T02:15:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014685304, 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 195 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322083+0000", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322084+0000", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322092+0000", 2024-09-16T02:15:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6529999999999994e-06 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322111+0000", 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8909000000000001e-05 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.322124+0000", 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3589e-05 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327750+0000", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056258819999999996 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.327883+0000", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013255499999999999 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.334789+0000", 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069061050000000001 2024-09-16T02:15:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.335013+0000", 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022373400000000001 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:27.336768+0000", 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001755454 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:504 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~786432] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.556469368000002, 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022327653999999999, 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 504 2024-09-16T02:15:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996033+0000", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1969999999999998e-06 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.997993+0000", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019604309999999999 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998029+0000", 2024-09-16T02:15:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5525999999999998e-05 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998044+0000", 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5440000000000001e-05 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998168+0000", 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012395 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998340+0000", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171619 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.000528+0000", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021878459999999998 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004445+0000", 2024-09-16T02:15:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003916966 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018230+0000", 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013784892 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018356+0000", 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126787 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:505 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~491520] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.549127683999998, 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022250066999999998, 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.574 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 505 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003374+0000", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.517e-06 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004387+0000", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001013445 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004415+0000", 2024-09-16T02:15:24.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004434+0000", 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9966000000000002e-05 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004556+0000", 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121734 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004705+0000", 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014862999999999999 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.006184+0000", 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014795570000000001 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.578 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.008703+0000", 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025187090000000001 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025506+0000", 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016802766 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025620+0000", 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114627 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:506 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~278528] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:24.580 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.542616944999999, 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016078973, 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 506 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009881+0000", 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.009892+0000", 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0849999999999999e-05 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010263+0000", 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037122700000000002 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010289+0000", 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5979e-05 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010303+0000", 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3876999999999999e-05 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010423+0000", 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119824 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.584 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010635+0000", 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000212091 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.011728+0000", 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001092927 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015127+0000", 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033989139999999998 2024-09-16T02:15:24.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025851+0000", 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01072415 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025960+0000", 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109134 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:507 3.8s0 3:177f2fba:::smithi19530025-45:head [setxattr _header (57) in=7b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.542163273, 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016008728999999999, 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.587 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 507 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010335+0000", 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010336+0000", 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.623e-06 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010345+0000", 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4500000000000004e-06 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.589 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010364+0000", 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9110000000000002e-05 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010379+0000", 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4676e-05 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.012892+0000", 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002513286 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.013156+0000", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026390700000000002 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014239+0000", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00108344 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.015432+0000", 2024-09-16T02:15:24.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001192961 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026211+0000", 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010778424999999999 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.026344+0000", 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013285099999999999 2024-09-16T02:15:24.592 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:508 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 37.542097534, 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019269792000000001, 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:24.593 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 508 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010401+0000", 2024-09-16T02:15:24.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010402+0000", 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3260000000000001e-06 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010408+0000", 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7699999999999998e-06 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010425+0000", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7603000000000001e-05 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.010443+0000", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8202999999999998e-05 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014784+0000", 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043404239999999998 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.014911+0000", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012705400000000001 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027153+0000", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012242116000000001 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.027274+0000", 2024-09-16T02:15:24.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012078099999999999 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.029670+0000", 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023965150000000001 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:586 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:24.598 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.643338918000001, 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018679077999999998, 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 586 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909159+0000", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909162+0000", 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.897e-06 2024-09-16T02:15:24.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909177+0000", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4630000000000001e-05 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909215+0000", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8588999999999999e-05 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909237+0000", 2024-09-16T02:15:24.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.211e-05 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909418+0000", 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000181003 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909440+0000", 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1271999999999999e-05 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.915451+0000", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006010858 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.917204+0000", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017530720000000001 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.920089+0000", 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002885267 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926620+0000", 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065311190000000002 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927778+0000", 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001157571 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927838+0000", 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0690000000000003e-05 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:587 3.8s0 3:165ce5f6:::smithi19530025-40:head [rollback d] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.643219355999999, 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023594436999999999, 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.605 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 587 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909280+0000", 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8129999999999998e-06 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909310+0000", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2731999999999999e-05 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909329+0000", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9134e-05 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909687+0000", 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035797500000000001 2024-09-16T02:15:24.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909698+0000", 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921510+0000", 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011812049999999999 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921788+0000", 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027776500000000002 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.924415+0000", 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00262737 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926686+0000", 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022710270000000001 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932811+0000", 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061244309999999996 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932873+0000", 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2630000000000002e-05 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:588 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.643115011999999, 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026094544000000001, 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 588 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909384+0000", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909391+0000", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6159999999999998e-06 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909409+0000", 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8335999999999999e-05 2024-09-16T02:15:24.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909424+0000", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.448e-05 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909774+0000", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035054599999999997 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909792+0000", 2024-09-16T02:15:24.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.77e-05 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925523+0000", 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015730929000000001 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925651+0000", 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012825900000000001 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926311+0000", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065990199999999997 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927500+0000", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001188599 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935426+0000", 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079261060000000005 2024-09-16T02:15:24.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935478+0000", 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1684000000000002e-05 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:598 3.8s0 3:1e2b38c5:::smithi19530025-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e69)", 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.611950910000001, 2024-09-16T02:15:24.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017113489999999999, 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 598 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:24.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940547+0000", 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940548+0000", 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940552+0000", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.507e-06 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940563+0000", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1177e-05 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.940571+0000", 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.763e-06 2024-09-16T02:15:24.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955266+0000", 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014695142 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.955563+0000", 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029711000000000001 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:57.957661+0000", 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002097921 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s5 72/66 ECSubWrite(tid=268, reqid=client.4598.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=72'21))", 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.592552468000001, 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51318274200000003, 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.623 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959947+0000", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959959+0000", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2097e-05 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471943+0000", 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51198411399999999 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471961+0000", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8641999999999999e-05 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472084+0000", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012220300000000001 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472100+0000", 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6583e-05 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473040+0000", 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094022500000000005 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473128+0000", 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7935000000000004e-05 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.627 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:654 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.595038956, 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020180719, 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 654 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957462+0000", 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7800000000000001e-06 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957473+0000", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1266e-05 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957499+0000", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5978000000000001e-05 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957519+0000", 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545999999999999e-05 2024-09-16T02:15:24.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959927+0000", 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024080640000000001 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:24.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029093199999999999 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962184+0000", 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019666520000000002 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.975499+0000", 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013314994 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977423+0000", 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019231490000000001 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977640+0000", 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021735799999999999 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:655 3.8s0 3:177f2fba:::smithi19530025-45:head [rollback d] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.594952262, 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023374418000000001, 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 655 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957547+0000", 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:15:24.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957551+0000", 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1659999999999996e-06 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957562+0000", 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0773e-05 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957570+0000", 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5520000000000005e-06 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965685+0000", 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081150130000000008 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965988+0000", 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030228299999999998 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968839+0000", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028510089999999998 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977674+0000", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088351860000000001 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980727+0000", 2024-09-16T02:15:24.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030533589999999998 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980920+0000", 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019316299999999999 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:656 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:24.639 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.594915883999999, 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034162017000000003, 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 656 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957583+0000", 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2600000000000002e-07 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957586+0000", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1870000000000001e-06 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957595+0000", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0189999999999995e-06 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957602+0000", 2024-09-16T02:15:24.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1629999999999999e-06 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972054+0000", 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452242000000001 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972340+0000", 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028568300000000001 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974084+0000", 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001744034 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977813+0000", 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037288239999999999 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991646+0000", 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013832947 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991744+0000", 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8292e-05 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:792 3.8s0 3:1cf09812:::smithi19530025-15:head [append 0~540672] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5479246480000004, 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022538648000000001, 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.645 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 792 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004576+0000", 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.486e-06 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005623+0000", 2024-09-16T02:15:24.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001047302 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005653+0000", 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9495000000000001e-05 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005668+0000", 2024-09-16T02:15:24.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4806e-05 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005825+0000", 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015700299999999999 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006087+0000", 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026202199999999999 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.008466+0000", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002379777 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.012994+0000", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045274139999999996 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.026934+0000", 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013940697 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.027112+0000", 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000177646 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:793 3.8s0 3:1cf09812:::smithi19530025-15:head [append 0~720896] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.014355+0000", 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5381426219999996, 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017932660999999999, 2024-09-16T02:15:24.652 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 793 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.014355+0000", 2024-09-16T02:15:24.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.014355+0000", 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.014359+0000", 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2169999999999999e-06 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.015952+0000", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015935540000000001 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.015978+0000", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5443e-05 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.015990+0000", 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2512e-05 2024-09-16T02:15:24.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.016205+0000", 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021499400000000001 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.016520+0000", 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031440299999999999 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.656 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.018555+0000", 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020357550000000002 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.020865+0000", 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023099269999999998 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.032115+0000", 2024-09-16T02:15:24.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011250031000000001 2024-09-16T02:15:24.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.032288+0000", 2024-09-16T02:15:24.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017282500000000001 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:796 3.8s0 3:1cf09812:::smithi19530025-15:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5302428900000002, 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.44076893700000003, 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.660 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 796 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022257+0000", 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022265+0000", 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1659999999999994e-06 2024-09-16T02:15:24.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022285+0000", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0225999999999999e-05 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022301+0000", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5458999999999999e-05 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.029702+0000", 2024-09-16T02:15:24.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074008829999999996 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.029842+0000", 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140848 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.030741+0000", 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089882099999999995 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.033161+0000", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024193819999999999 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.462883+0000", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.42972244900000001 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.463024+0000", 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000141063 2024-09-16T02:15:24.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:797 3.8s0 3:1cf09812:::smithi19530025-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.5301760929999997, 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.444497743, 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 797 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.667 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:24.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022323+0000", 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4330000000000001e-06 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022330+0000", 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4169999999999997e-06 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022350+0000", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9729e-05 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022364+0000", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4735999999999999e-05 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.033962+0000", 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011597481999999999 2024-09-16T02:15:24.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.034096+0000", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013378399999999999 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.463960+0000", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.42986474299999999 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.464088+0000", 2024-09-16T02:15:24.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000127537 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.466820+0000", 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027318820000000001 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 91/69 ECSubWrite(tid=183, reqid=osd.11.0:183, at_version=91'176, trim_to=0'0, roll_forward_to=91'170))", 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:24.672 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.5754018710000004, 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85248054399999995, 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:24.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977098+0000", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7859999999999999e-06 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977135+0000", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6545999999999999e-05 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827628+0000", 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.850493583 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827644+0000", 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855999999999999e-05 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827758+0000", 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114369 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827782+0000", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3289e-05 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.829489+0000", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017076529999999999 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.829577+0000", 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7461999999999997e-05 2024-09-16T02:15:24.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.678 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.678 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:24.678 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:24.678 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:24.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:24.681 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:24.678+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:24.780 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:24.776+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:24.839 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 5 to smithi19530025-47 2024-09-16T02:15:24.842 INFO:tasks.rados.rados.0.smithi195.stdout:340: finishing write tid 6 to smithi19530025-47 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: oid 47 updating version 0 to 35 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: oid 47 version 35 is already newer than 33 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: oid 47 version 35 is already newer than 34 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: oid 47 version 35 is already newer than 32 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: oid 47 version 35 is already newer than 31 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 47 v 35 (ObjNum 134 snap 31 seq_num 134) dirty exists 2024-09-16T02:15:24.843 INFO:tasks.rados.rados.0.smithi195.stdout:340: left oid 47 (ObjNum 134 snap 31 seq_num 134) 2024-09-16T02:15:24.853 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:24.853 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:24.853 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:24.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:24.911 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:24.908+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:24.982 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:24.982+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:24.986 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:338: done (4 left) 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:339: done (3 left) 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:340: done (2 left) 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:341: done (1 left) 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:342: done (0 left) 2024-09-16T02:15:24.995 INFO:tasks.rados.rados.0.smithi195.stdout:343: snap_create 2024-09-16T02:15:25.035 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:25.036 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:25.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:25.099 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:25.096+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.892263794999998, 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.888041348999998, 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.887746040000003, 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006345626, 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283858+0000", 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283859+0000", 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283921+0000", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1947999999999997e-05 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284076+0000", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015468599999999999 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284095+0000", 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8686e-05 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285435+0000", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001340639 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285464+0000", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8411999999999998e-05 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290127+0000", 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046638529999999999 2024-09-16T02:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290204+0000", 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079000000000001e-05 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.875289416999998, 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.872625036999999, 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.872565817000002, 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.872158511999999, 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.870277766999997, 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.870002202999999, 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 60.868676806000003, 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00583893, 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302927+0000", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302929+0000", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.717e-06 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302970+0000", 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1564999999999999e-05 2024-09-16T02:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303048+0000", 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8059000000000006e-05 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303070+0000", 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1282999999999999e-05 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306196+0000", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00312663 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306213+0000", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6747000000000001e-05 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308730+0000", 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002516796 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308766+0000", 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6133000000000001e-05 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=45, reqid=client.4598.0:244, at_version=44'36, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.752138973000001, 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013820059000000001, 2024-09-16T02:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419467+0000", 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9400000000000001e-06 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419693+0000", 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000226312 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431387+0000", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011694154999999999 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431404+0000", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6945000000000001e-05 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431560+0000", 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015586199999999999 2024-09-16T02:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431575+0000", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4979e-05 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433182+0000", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016072700000000001 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433285+0000", 2024-09-16T02:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010259600000000001 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=46, reqid=client.4598.0:245, at_version=44'38, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.740521858999998, 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060040980000000002, 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431082+0000", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431084+0000", 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.823e-06 2024-09-16T02:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431346+0000", 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026225100000000002 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434399+0000", 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030528320000000001 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434422+0000", 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3326e-05 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434546+0000", 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411200000000001 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.434562+0000", 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5075999999999999e-05 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437029+0000", 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024676699999999999 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.437086+0000", 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7008e-05 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=48, reqid=client.4598.0:247, at_version=44'40, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.736582765000001, 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29780827999999998, 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435023+0000", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435063+0000", 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:15:25.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731424+0000", 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29636148400000001 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731439+0000", 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.453e-05 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731595+0000", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015660699999999999 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731614+0000", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8243000000000001e-05 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732758+0000", 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011446360000000001 2024-09-16T02:15:25.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732829+0000", 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1191000000000003e-05 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.814709897, 2024-09-16T02:15:25.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080490270000000003, 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:25.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356895+0000", 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356948+0000", 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3032000000000003e-05 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357075+0000", 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012766899999999999 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357100+0000", 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5086e-05 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357282+0000", 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018186299999999999 2024-09-16T02:15:25.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7348999999999999e-05 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364849+0000", 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075397019999999997 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364943+0000", 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3709999999999996e-05 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.809958504000001, 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070518059999999999, 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.238 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361646+0000", 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361693+0000", 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6882000000000003e-05 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361805+0000", 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112046 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361834+0000", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9215999999999999e-05 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362268+0000", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043351899999999999 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362294+0000", 2024-09-16T02:15:25.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6142000000000001e-05 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368607+0000", 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063134869999999996 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9844000000000002e-05 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.242 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 50.807883588000003, 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059443869999999998, 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363720+0000", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363722+0000", 2024-09-16T02:15:25.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.733e-06 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363805+0000", 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2776999999999997e-05 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363906+0000", 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010156 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.363931+0000", 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4966000000000001e-05 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365907+0000", 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019751790000000001 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.365933+0000", 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205999999999999e-05 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369569+0000", 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036364890000000001 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.369665+0000", 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5476999999999994e-05 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 91/69 ECSubWrite(tid=176, reqid=client.4598.0:812, at_version=91'166, trim_to=0'0, roll_forward_to=90'164))", 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.955623+0000", 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.2159806440000001, 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058125420000000004, 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.248 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.955623+0000", 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.955623+0000", 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.955624+0000", 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0999999999999998e-07 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956374+0000", 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074991700000000001 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956462+0000", 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8475999999999998e-05 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956489+0000", 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6222000000000001e-05 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956666+0000", 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017762600000000001 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956691+0000", 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5228999999999998e-05 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.961340+0000", 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046491060000000001 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.961436+0000", 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5356e-05 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 91/32 ECSubWrite(tid=46, reqid=client.4598.0:825, at_version=91'27, trim_to=0'0, roll_forward_to=84'25))", 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:25.253 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1736011179999997, 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063829200000000003, 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998004+0000", 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5600000000000004e-07 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998976+0000", 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097227800000000003 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001818+0000", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028420189999999999 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001830+0000", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.241e-05 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.002052+0000", 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221108 2024-09-16T02:15:25.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.002067+0000", 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5033e-05 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004336+0000", 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022693420000000001 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004386+0000", 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9774000000000002e-05 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 91/32 ECSubWrite(tid=48, reqid=client.4598.0:827, at_version=91'29, trim_to=0'0, roll_forward_to=84'25))", 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:20.004849+0000", 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1667551769999998, 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057623350000000004, 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.258 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004849+0000", 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004849+0000", 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004851+0000", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9130000000000001e-06 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.005131+0000", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000280518 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.007852+0000", 2024-09-16T02:15:25.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027203850000000001 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.007876+0000", 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4226000000000001e-05 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.007991+0000", 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115096 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.008004+0000", 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3538e-05 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.010565+0000", 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002560489 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.010611+0000", 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.617e-05 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 91/32 ECSubWrite(tid=51, reqid=client.4598.0:831, at_version=91'33, trim_to=0'0, roll_forward_to=91'28))", 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 5.1622206750000004, 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038355847999999998, 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.263 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009385+0000", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009414+0000", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9739000000000001e-05 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.045925+0000", 2024-09-16T02:15:25.265 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036510501000000001 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.045940+0000", 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5366000000000002e-05 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.046104+0000", 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016406299999999999 2024-09-16T02:15:25.266 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.046141+0000", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6924000000000001e-05 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.047694+0000", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015532969999999999 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.047739+0000", 2024-09-16T02:15:25.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4635000000000002e-05 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.268 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:25.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:25.272 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:25.266+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:25.370 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:25.364+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:25.437 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:25.437 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:25.437 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:25.437 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:25.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:25.495 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:25.492+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:25.571 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:25.568+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:25.616 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:25.616 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:25.616 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:25.616 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:25.616 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:25.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:25.675 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:25.672+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:25.772 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:25.769+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:25.801 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:25.801 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:25.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:25.801 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 79.726656993999995, 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:25.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:15:25.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:15:25.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.888801189000006, 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022495154, 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:25.809 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 123 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909118+0000", 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909119+0000", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7700000000000004e-07 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909204+0000", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4952999999999998e-05 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909218+0000", 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4188999999999999e-05 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909226+0000", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2430000000000005e-06 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923390+0000", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01416416 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923611+0000", 2024-09-16T02:15:25.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220757 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.923971+0000", 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035964299999999999 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926701+0000", 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027305680000000001 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931491+0000", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004789916 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931560+0000", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8602000000000005e-05 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.931613+0000", 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3346000000000001e-05 2024-09-16T02:15:25.814 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 78.888676794999995, 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.815 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:15:25.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:15:25.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:1 3.3s0 3:c481b30a:::smithi19530025-1:head [append 0~32768,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.520524158000001, 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021556188, 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277395+0000", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277399+0000", 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4400000000000001e-06 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277527+0000", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012811100000000001 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277557+0000", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0645000000000002e-05 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277594+0000", 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6848999999999999e-05 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277805+0000", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021101299999999999 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.278199+0000", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039380300000000002 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.280065+0000", 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018661680000000001 2024-09-16T02:15:25.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286512+0000", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064472219999999998 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298756+0000", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012244026 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298951+0000", 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000194911 2024-09-16T02:15:25.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:2 3.3s0 3:c481b30a:::smithi19530025-1:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.520270289000003, 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022809754000000002, 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.827 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277649+0000", 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277650+0000", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277657+0000", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277675+0000", 2024-09-16T02:15:25.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8148999999999999e-05 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277691+0000", 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5943e-05 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282023+0000", 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004332247 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282308+0000", 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028409399999999998 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.285666+0000", 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033580379999999998 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.292774+0000", 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071080379999999997 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300235+0000", 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074610759999999996 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300459+0000", 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224033 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:25.832 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.520105616999999, 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:15:25.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.837 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.516627223999997, 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:15:25.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs6 35/32 ECSubWrite(tid=3, reqid=client.4598.0:20, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.515505050999998, 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020853072, 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282414+0000", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282416+0000", 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:25.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282453+0000", 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7261000000000001e-05 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298874+0000", 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016420846999999999 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298893+0000", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9695e-05 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299501+0000", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000607509 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299522+0000", 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1039e-05 2024-09-16T02:15:25.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303194+0000", 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036718020000000001 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303267+0000", 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3598999999999998e-05 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.514984214999998, 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.850 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:15:25.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:15:25.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.856 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.514887995999999, 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:25.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:15:25.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:58 3.2s0 3:431ee5ca:::smithi19530025-20:head [append 0~327680,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.482858641999997, 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021031579000000002, 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.863 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 58 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315061+0000", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315062+0000", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315685+0000", 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062327200000000002 2024-09-16T02:15:25.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315711+0000", 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5519000000000001e-05 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315731+0000", 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9775e-05 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.322701+0000", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069698679999999997 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.323044+0000", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034300800000000001 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.324822+0000", 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017778659999999999 2024-09-16T02:15:25.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.329203+0000", 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043815149999999999 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.335988+0000", 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067852160000000002 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.336092+0000", 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000103927 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:59 3.2s0 3:431ee5ca:::smithi19530025-20:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.482148741000003, 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022592232, 2024-09-16T02:15:25.869 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 59 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315771+0000", 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8729999999999999e-06 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315782+0000", 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2629999999999992e-06 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315822+0000", 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0095000000000002e-05 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315843+0000", 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0905999999999999e-05 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.326893+0000", 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011050249999999999 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.327039+0000", 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000145723 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328011+0000", 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097276899999999996 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.330713+0000", 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027020989999999999 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338142+0000", 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074284329999999999 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.338363+0000", 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022082100000000001 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.482049574999998, 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:15:25.875 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:25.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:15:25.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.880 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.430313828000003, 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:15:25.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:15:25.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 51.426026147999998, 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.885 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:15:25.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.731471585000001, 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27886435399999998, 2024-09-16T02:15:25.889 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:25.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066449+0000", 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.435e-06 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066471+0000", 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1526999999999999e-05 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343774+0000", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27730348599999999 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343789+0000", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4946000000000001e-05 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343981+0000", 2024-09-16T02:15:25.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019161199999999999 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.344000+0000", 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545e-05 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345225+0000", 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012244479999999999 2024-09-16T02:15:25.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345312+0000", 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7355000000000006e-05 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 84/83 ECSubWriteReply(tid=304, last_complete=84'106, committed=1, applied=1))", 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:25.894 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.634363050999999, 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67482556900000001, 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163558+0000", 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163565+0000", 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5129999999999999e-06 2024-09-16T02:15:25.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837152+0000", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67358686899999998 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837166+0000", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3685e-05 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837280+0000", 2024-09-16T02:15:25.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114374 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.838382+0000", 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011018079999999999 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:755 3.2s0 3:42c84d65:::smithi19530025-36:head [read 80856~17448,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.065260+0000", 2024-09-16T02:15:25.898 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.732659698999999, 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.021834280000000001, 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 755 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065260+0000", 2024-09-16T02:15:25.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065260+0000", 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065261+0000", 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6130000000000001e-06 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065271+0000", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3759999999999997e-06 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065291+0000", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0125000000000001e-05 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065305+0000", 2024-09-16T02:15:25.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4606999999999999e-05 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065817+0000", 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00051172700000000002 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.065836+0000", 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9443e-05 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.079560+0000", 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013723915 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.079788+0000", 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227405 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.087094+0000", 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073060690000000001 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:756 3.2s0 3:49354246:::smithi19530025-34:head [create,setxattr Ytb`ShRqaiPs[rr (16840) in=15b,setxattr q^\\`^ttfZZ[onOQp[Nk_jhVlRR[i\\U^PogkqYo]p (1857) in=40b,setxattr l\\_QQkdTct[dh_kh]ZhkpR[QfYlnOOdpYPrn_RjY (34) in=40b,setxattr Oq^VPc^pYslWd]Q[ceVVgpbYn^o`q^^l[gchYaYN (1998) in=40b,setxattr [StfOUatjPeRjmq[saNOZkP`ZqfbasRnQgNO_e`[ (1613) in=40b,setxattr `cnSsV[oaPnjPoojaS`jkbmjeaOa[WZY\\SgW]UPk (1894) in=40b,setxattr ON\\^PP_lXNb^Nq^ldrgr]_]XZdqY_`Y`^UjcaP_T (1506) in=40b,setxattr sna_UWhn`^XSjWgrctk`OmcqtNnapNjQqUi\\[eSp (1189) in=40b,setxattr h[m]qgpf^]k]`s (16840) in=14b,setxattr il]WmarWmWifQtaTrVd^ZTPX[i\\P\\cTiPZ]YTVdR (1189) in=40b,setxattr hatXYlRa[e\\odOmcXqd\\VrY\\eY\\f^]kTS_OpPQPQ (34) in=40b,setxattr Oa_afbXaQtgVaUpZ[[QbfTScP[gY]hcXaRdQ[_Vk (19508) in=40b,setxattr Q\\cPn^s\\XTid_Yg_O\\niYXXidVnUdqsUOU`d_cWU (1859) in=40b,setxattr ^aROPrYY^[s\\UgNbafm^etaYNpV_^kpTRhgcaqjX (663) in=40b,setxattr jcqcjeaa`t]VPoaXgmkenU``eQtX[Vafoofs]`^X (1962) in=40b,setxattr giUUaha_tnig_N`i_nVNiWf[R]n^T[hSX]tn^mmX (1927) in=40b,setxattr bSbnrc_VRfjaSV`VNt```Y`[gXiTeZNVQiYTml\\\\ (12488) in=40b,setxattr efUSdinatVQlXejsl]j^UbZ\\VggQpZWYknRamcrN (383) in=40b,setxattr jP_Q_QrSjsgdjt`ORsRsZdpQoaUhTWQtj_Q_qplV (16807) in=40b,setxattr as^Pblt]tYZ\\ZO_tmgVg_YedlfbaXOaZ[jgOZQt` (561) in=40b] snapc 19=[18,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.075651+0000", 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.722268340999999, 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025428531000000001, 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.904 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 756 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075651+0000", 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075651+0000", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075655+0000", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7129999999999999e-06 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.075965+0000", 2024-09-16T02:15:25.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031023200000000002 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.076005+0000", 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9657999999999999e-05 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.076030+0000", 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5891999999999999e-05 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.087206+0000", 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011175179 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.087760+0000", 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055471199999999996 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.090885+0000", 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003124876 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.097691+0000", 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068057760000000004 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.100786+0000", 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00309487 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.101079+0000", 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029362300000000002 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.704487774, 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59632199500000005, 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:25.910 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093434+0000", 2024-09-16T02:15:25.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.39e-06 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093480+0000", 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5844999999999999e-05 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688435+0000", 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59495508799999997 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688450+0000", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5345999999999998e-05 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688617+0000", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166608 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688648+0000", 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1282000000000003e-05 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689677+0000", 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010285520000000001 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689753+0000", 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6884000000000005e-05 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:25.914 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:25.916 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:25.916 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:25.913+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:25.922 INFO:tasks.rados.rados.0.smithi195.stdout:343: done (0 left) 2024-09-16T02:15:25.922 INFO:tasks.rados.rados.0.smithi195.stdout:344: snap_remove snap 25 2024-09-16T02:15:26.017 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:26.013+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:26.044 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:26.044 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:26.044 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:26.044 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:26.058 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:26.118 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:26.114+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:26.218 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:26.218+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:26.223 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:26.223 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:26.223 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:26.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:26.319 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:26.314+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:26.366 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:26.366 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:26.366 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:26.366 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.173036250999999, 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:15:26.367 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.368 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.369 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:15:26.370 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.371 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 76.188010014, 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066455200000000002, 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 7 2024-09-16T02:15:26.372 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175558+0000", 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.373 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175561+0000", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7030000000000002e-06 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175575+0000", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3889e-05 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175608+0000", 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3704000000000001e-05 2024-09-16T02:15:26.374 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175627+0000", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175763+0000", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013625000000000001 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.175774+0000", 2024-09-16T02:15:26.375 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.116e-05 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179478+0000", 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037039989999999999 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179657+0000", 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179538 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for subops from 7", 2024-09-16T02:15:26.376 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.179882+0000", 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022455700000000001 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op_commit", 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.181213+0000", 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013309039999999999 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182065+0000", 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00085171900000000002 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.377 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182143+0000", 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.7904999999999994e-05 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:10.182203+0000", 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.0909000000000002e-05 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 38/32 ECSubWrite(tid=14, reqid=client.4598.0:197, at_version=38'9, trim_to=0'0, roll_forward_to=35'6))", 2024-09-16T02:15:26.378 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 59.036427652, 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058517350000000003, 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.379 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327140+0000", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327142+0000", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.59e-06 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327187+0000", 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.5467999999999999e-05 2024-09-16T02:15:26.380 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327289+0000", 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102086 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327312+0000", 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2849000000000001e-05 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.381 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327454+0000", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014216700000000001 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327467+0000", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3009e-05 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332903+0000", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054355560000000002 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.382 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.332992+0000", 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9010000000000003e-05 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.020444818999998, 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095323660000000004, 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.383 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343124+0000", 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.06e-06 2024-09-16T02:15:26.384 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343143+0000", 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8842e-05 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343322+0000", 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179414 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.385 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343345+0000", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2629e-05 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343569+0000", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022398900000000001 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343596+0000", 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6489e-05 2024-09-16T02:15:26.386 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352536+0000", 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089404250000000001 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352656+0000", 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119518 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.387 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/32 ECSubWrite(tid=19, reqid=osd.6.0:19, at_version=44'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.020355572, 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083893449999999994, 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.388 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343214+0000", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343264+0000", 2024-09-16T02:15:26.389 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0175e-05 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343378+0000", 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114159 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343396+0000", 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7926000000000001e-05 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.390 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343513+0000", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117056 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343528+0000", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4324999999999999e-05 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351501+0000", 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079735710000000005 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.391 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351602+0000", 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010059 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 53.019020660999999, 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082631800000000002, 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.392 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.393 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344549+0000", 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344608+0000", 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8953999999999998e-05 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.394 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344655+0000", 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7268999999999998e-05 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344677+0000", 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1382e-05 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344840+0000", 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016295099999999999 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.395 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344865+0000", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5035e-05 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352714+0000", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078496989999999999 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352810+0000", 2024-09-16T02:15:26.396 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6146999999999999e-05 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.016205315000001, 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009462359, 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.397 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347364+0000", 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.623e-06 2024-09-16T02:15:26.398 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347386+0000", 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1667000000000001e-05 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347450+0000", 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4116999999999996e-05 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.399 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347464+0000", 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4347e-05 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348099+0000", 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063462999999999996 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348122+0000", 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2866000000000001e-05 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.400 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356742+0000", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086204220000000008 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356825+0000", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2687000000000003e-05 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.9s6 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:26.401 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.014482725999997, 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0076692430000000001, 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349085+0000", 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.402 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349127+0000", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1560999999999997e-05 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349203+0000", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6447999999999997e-05 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349306+0000", 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102413 2024-09-16T02:15:26.403 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349329+0000", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3618999999999999e-05 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349580+0000", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000250874 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.349604+0000", 2024-09-16T02:15:26.404 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4253e-05 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356660+0000", 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070559610000000004 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356754+0000", 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4114e-05 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.405 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 45/32 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=45'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.012295436999999, 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010034855000000001, 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.406 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351322+0000", 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.407 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351430+0000", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108572 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351458+0000", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8229e-05 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351962+0000", 2024-09-16T02:15:26.408 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050377899999999999 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351989+0000", 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6322e-05 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361119+0000", 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091308310000000007 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.409 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361307+0000", 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188054 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 51/32 ECSubWrite(tid=41, reqid=osd.6.0:41, at_version=51'32, trim_to=0'0, roll_forward_to=49'30))", 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.949964551999997, 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0075541130000000003, 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.410 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413603+0000", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413604+0000", 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.411 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.413653+0000", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8726000000000001e-05 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414322+0000", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00066915500000000005 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414345+0000", 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2863000000000001e-05 2024-09-16T02:15:26.412 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414490+0000", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014519199999999999 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414515+0000", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5312000000000001e-05 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421073+0000", 2024-09-16T02:15:26.413 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065573719999999997 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.421158+0000", 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.4690000000000004e-05 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:394 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.456247476999998, 2024-09-16T02:15:26.414 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094620629999999997, 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 394 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.415 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907323+0000", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5399999999999998e-06 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907336+0000", 2024-09-16T02:15:26.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2605999999999999e-05 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907364+0000", 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8538999999999999e-05 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907387+0000", 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979000000000001e-05 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907499+0000", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011208 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907666+0000", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016685800000000001 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.909169+0000", 2024-09-16T02:15:26.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015026659999999999 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914888+0000", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057194580000000002 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916546+0000", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001657911 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916783+0000", 2024-09-16T02:15:26.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236426 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:395 3.fs0 3:f97ec54c:::smithi19530025-26:head [rollback 8] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.456143167, 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011190179, 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:26.420 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 395 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:26.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907426+0000", 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907433+0000", 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907453+0000", 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907485+0000", 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2066000000000001e-05 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910164+0000", 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026789510000000002 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910300+0000", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013539300000000001 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.911564+0000", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001264577 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.917774+0000", 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062099729999999997 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918506+0000", 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073229000000000002 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918615+0000", 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108527 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:396 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:26.425 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 45.456057113, 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011749021, 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 396 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:26.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907512+0000", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1060000000000001e-06 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907519+0000", 2024-09-16T02:15:26.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.917e-06 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907537+0000", 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907552+0000", 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912502+0000", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049500600000000001 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912741+0000", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023893499999999999 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914057+0000", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013153119999999999 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918884+0000", 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048277240000000003 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919157+0000", 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306100000000001 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919260+0000", 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102594 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.402269089000001, 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0067590630000000001, 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961299+0000", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961300+0000", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.8999999999999995e-07 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961666+0000", 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036643500000000002 2024-09-16T02:15:26.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961786+0000", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120192 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.961811+0000", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4405000000000001e-05 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962023+0000", 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000212489 2024-09-16T02:15:26.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.962050+0000", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6968e-05 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.967810+0000", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057602699999999996 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.968058+0000", 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000247514 2024-09-16T02:15:26.434 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.371722387999998, 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091235069999999995, 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991847+0000", 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0699999999999999e-06 2024-09-16T02:15:26.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991900+0000", 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3204000000000001e-05 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992025+0000", 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012476300000000001 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992053+0000", 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8085e-05 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992229+0000", 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017621299999999999 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992255+0000", 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6319e-05 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000887+0000", 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086322410000000006 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000969+0000", 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1612000000000004e-05 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/32 ECSubWrite(tid=97, reqid=osd.6.0:97, at_version=79'82, trim_to=0'0, roll_forward_to=78'80))", 2024-09-16T02:15:26.439 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:07.993890+0000", 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.369678188999998, 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065695570000000002, 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.993890+0000", 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.993890+0000", 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.993891+0000", 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6000000000000003e-07 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.993959+0000", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7999999999999999e-05 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.994079+0000", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000120266 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.994097+0000", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8176e-05 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.994273+0000", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000176133 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.994313+0000", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9771000000000003e-05 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000377+0000", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060644460000000002 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000459+0000", 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2004999999999998e-05 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 79/32 ECSubWrite(tid=98, reqid=osd.6.0:98, at_version=79'84, trim_to=0'0, roll_forward_to=78'80))", 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:07.997365+0000", 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.366202513000001, 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0066568260000000002, 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.444 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997365+0000", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997365+0000", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997367+0000", 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8700000000000001e-06 2024-09-16T02:15:26.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997425+0000", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8119999999999999e-05 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997532+0000", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010657700000000001 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997555+0000", 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3238999999999999e-05 2024-09-16T02:15:26.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997703+0000", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014731400000000001 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.997729+0000", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5945e-05 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.003937+0000", 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062087089999999998 2024-09-16T02:15:26.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.004022+0000", 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.5051999999999998e-05 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(osd.7.22:3 3.fs0 3:f2f6c7ec:::smithi19530025-24:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.322735388000002, 2024-09-16T02:15:26.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090619859999999993, 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040836+0000", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4659999999999999e-06 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040855+0000", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8689e-05 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040906+0000", 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1381000000000002e-05 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040930+0000", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3739000000000001e-05 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041122+0000", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191669 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041284+0000", 2024-09-16T02:15:26.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000162917 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.049895+0000", 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086101249999999997 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 85/66 ECSubWrite(tid=171, reqid=client.4598.0:760, at_version=85'57, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:14.080365+0000", 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.283203445, 2024-09-16T02:15:26.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0065997449999999997, 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080365+0000", 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080365+0000", 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080371+0000", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5560000000000004e-06 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080858+0000", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048686399999999999 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080968+0000", 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011049500000000001 2024-09-16T02:15:26.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.080991+0000", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2532e-05 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.081134+0000", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014296600000000001 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.081149+0000", 2024-09-16T02:15:26.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5420000000000001e-05 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.086870+0000", 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057210960000000002 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.086964+0000", 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3815999999999998e-05 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 12.270077401, 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50299762999999997, 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093493+0000", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2400000000000002e-06 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093538+0000", 2024-09-16T02:15:26.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4971e-05 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595224+0000", 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50168629200000003 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595240+0000", 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.56e-05 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:26.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595366+0000", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012640000000000001 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595382+0000", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5936000000000001e-05 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596420+0000", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010379499999999999 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:26.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596488+0000", 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8240999999999999e-05 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:26.461 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:26.462 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:26.463 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:26.458+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:26.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:26.563 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:26.561+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:26.583 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:26.584 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:26.584 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:26.584 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:26.597 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:26.638 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:26.638 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:26.664 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:26.662+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:26.666 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:26.666 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:26.695 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:26.695 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:26.727 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:26.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:26.758 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:26.758 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:26.765 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:26.765 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:26.765 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:26.765 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:26.765 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:26.766 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:26.760+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:26.779 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:26.792 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:26.793 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:26.823 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:26.824 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:26.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:26.859 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:26.867 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:26.866+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:26.892 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:26.892 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:26.897 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:26.897 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:26.912 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:26.922 INFO:tasks.rados.rados.0.smithi195.stdout:344: done (0 left) 2024-09-16T02:15:26.922 INFO:tasks.rados.rados.0.smithi195.stdout:345: copy_from oid 27 from oid 39 current snap is 33 2024-09-16T02:15:26.922 INFO:tasks.rados.rados.0.smithi195.stdout:346: read oid 38 snap -1 2024-09-16T02:15:26.922 INFO:tasks.rados.rados.0.smithi195.stdout:346: expect deleted 2024-09-16T02:15:26.922 INFO:tasks.rados.rados.0.smithi195.stdout:347: delete oid 13 current snap is 33 2024-09-16T02:15:26.930 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:26.930 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:26.930 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-09-16T02:15:26.954 INFO:tasks.rados.rados.0.smithi195.stdout:346: done (2 left) 2024-09-16T02:15:26.954 INFO:tasks.rados.rados.0.smithi195.stdout:347: done (1 left) 2024-09-16T02:15:26.954 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 12 current snap is 33 2024-09-16T02:15:26.954 INFO:tasks.rados.rados.0.smithi195.stdout:348: seq_num 136 ranges {196608=737280,933888=770048,1703936=540672,2244608=245760} 2024-09-16T02:15:26.966 INFO:tasks.rados.rados.0.smithi195.stdout:348: writing smithi19530025-12 from 196608 to 933888 tid 1 2024-09-16T02:15:26.967 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:26.965+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:26.968 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:26.968 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:26.975 INFO:tasks.rados.rados.0.smithi195.stdout:348: writing smithi19530025-12 from 933888 to 1703936 tid 2 2024-09-16T02:15:26.981 INFO:tasks.rados.rados.0.smithi195.stdout:348: writing smithi19530025-12 from 1703936 to 2244608 tid 3 2024-09-16T02:15:26.984 INFO:tasks.rados.rados.0.smithi195.stdout:348: writing smithi19530025-12 from 2244608 to 2490368 tid 4 2024-09-16T02:15:26.984 INFO:tasks.rados.rados.0.smithi195.stdout:349: snap_remove snap 11 2024-09-16T02:15:26.994 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:26.995 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:27.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:27.019 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:27.020 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:27.044 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:27.045 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:27.047 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.048 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.062 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:27.068 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:27.064+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:27.070+0000 7f51f2ffd640 0 -- 172.21.15.3:0/646088739 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f51fc2226b0 msgr2=0x7f51fc224aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:27.082 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-16T02:15:27.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-16T02:15:27.110 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:27.109+0000 7f2af03cb7c0 -1 osd.10 81 log_to_monitors true 2024-09-16T02:15:27.169 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:27.164+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:27.217 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:27.224 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.224 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.238 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:27.270 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:27.268+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:27.334 INFO:teuthology.orchestra.run.smithi177.stderr:no valid command found; 10 closest matches: 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:0 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:1 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:2 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:abort 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:assert 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:bluefs debug_inject_read_zeros 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:bluefs files list 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:bluefs stats 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:bluestore allocator dump block 2024-09-16T02:15:27.335 INFO:teuthology.orchestra.run.smithi177.stderr:bluestore allocator fragmentation block 2024-09-16T02:15:27.336 INFO:teuthology.orchestra.run.smithi177.stderr:admin_socket: invalid command 2024-09-16T02:15:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-16T02:15:27.337 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-10, ['dump_ops_in_flight'] 2024-09-16T02:15:27.371 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:27.370+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:27.411 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.411 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.425 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:27.472 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:27.469+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:27.556 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.556 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.570 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:27.572 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:27.569+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:27.673 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:27.670+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:27.693 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.693 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.707 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:27.774 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:27.773+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:27.830 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.830 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.847 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:27.875 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:27.874+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:27.920 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:27.927 INFO:tasks.rados.rados.0.smithi195.stdout:349: done (2 left) 2024-09-16T02:15:27.927 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 2 current snap is 33 2024-09-16T02:15:27.927 INFO:tasks.rados.rados.0.smithi195.stdout:350: seq_num 137 ranges {409600=753664,1163264=786432,1949696=655360,2605056=114688} 2024-09-16T02:15:27.938 INFO:tasks.rados.rados.0.smithi195.stdout:350: writing smithi19530025-2 from 409600 to 1163264 tid 1 2024-09-16T02:15:27.947 INFO:tasks.rados.rados.0.smithi195.stdout:350: writing smithi19530025-2 from 1163264 to 1949696 tid 2 2024-09-16T02:15:27.953 INFO:tasks.rados.rados.0.smithi195.stdout:350: writing smithi19530025-2 from 1949696 to 2605056 tid 3 2024-09-16T02:15:27.955 INFO:tasks.rados.rados.0.smithi195.stdout:350: writing smithi19530025-2 from 2605056 to 2719744 tid 4 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:345: finishing copy_from racing read to smithi19530025-27 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:345: finishing copy_from to smithi19530025-27 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 27 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 1 to smithi19530025-12 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 2 to smithi19530025-12 2024-09-16T02:15:27.956 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 3 to smithi19530025-12 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 4 to smithi19530025-12 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 5 to smithi19530025-12 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: finishing write tid 6 to smithi19530025-12 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: oid 12 updating version 0 to 38 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: oid 12 version 38 is already newer than 36 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: oid 12 updating version 38 to 39 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: oid 12 version 39 is already newer than 37 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: oid 12 version 39 is already newer than 35 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 12 v 39 (ObjNum 136 snap 33 seq_num 136) dirty exists 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: left oid 12 (ObjNum 136 snap 33 seq_num 136) 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:345: done (2 left) 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:348: done (1 left) 2024-09-16T02:15:27.957 INFO:tasks.rados.rados.0.smithi195.stdout:351: read oid 28 snap -1 2024-09-16T02:15:27.958 INFO:tasks.rados.rados.0.smithi195.stdout:351: expect (ObjNum 90 snap 12 seq_num 90) 2024-09-16T02:15:27.958 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 10 current snap is 33 2024-09-16T02:15:27.958 INFO:tasks.rados.rados.0.smithi195.stdout:352: seq_num 138 ranges {1458176=770048,2228224=540672,2768896=753664,3522560=262144} 2024-09-16T02:15:27.962 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:27.961+0000 7f2ae81ae640 -1 osd.10 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T02:15:27.975 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:27.970+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:27.975 INFO:tasks.rados.rados.0.smithi195.stdout:352: writing smithi19530025-10 from 1458176 to 2228224 tid 1 2024-09-16T02:15:27.980 INFO:tasks.rados.rados.0.smithi195.stdout:352: writing smithi19530025-10 from 2228224 to 2768896 tid 2 2024-09-16T02:15:27.986 INFO:tasks.rados.rados.0.smithi195.stdout:352: writing smithi19530025-10 from 2768896 to 3522560 tid 3 2024-09-16T02:15:27.988 INFO:tasks.rados.rados.0.smithi195.stdout:352: writing smithi19530025-10 from 3522560 to 3784704 tid 4 2024-09-16T02:15:27.988 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 33 current snap is 33 2024-09-16T02:15:27.988 INFO:tasks.rados.rados.0.smithi195.stdout:353: seq_num 139 ranges {0=786432,786432=688128,1474560=475136,1949696=393216} 2024-09-16T02:15:27.990 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:27.990 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:27.993 INFO:tasks.rados.rados.0.smithi195.stdout:353: writing smithi19530025-33 from 0 to 786432 tid 1 2024-09-16T02:15:27.997 INFO:tasks.rados.rados.0.smithi195.stdout:353: writing smithi19530025-33 from 786432 to 1474560 tid 2 2024-09-16T02:15:28.000 INFO:tasks.rados.rados.0.smithi195.stdout:353: writing smithi19530025-33 from 1474560 to 1949696 tid 3 2024-09-16T02:15:28.003 INFO:tasks.rados.rados.0.smithi195.stdout:353: writing smithi19530025-33 from 1949696 to 2342912 tid 4 2024-09-16T02:15:28.003 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 22 current snap is 33 2024-09-16T02:15:28.003 INFO:tasks.rados.rados.0.smithi195.stdout:354: seq_num 140 ranges {360448=802816,1163264=425984,1589248=720896,2310144=409600} 2024-09-16T02:15:28.008 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:28.010 INFO:tasks.rados.rados.0.smithi195.stdout:354: writing smithi19530025-22 from 360448 to 1163264 tid 1 2024-09-16T02:15:28.012 INFO:tasks.rados.rados.0.smithi195.stdout:354: writing smithi19530025-22 from 1163264 to 1589248 tid 2 2024-09-16T02:15:28.017 INFO:tasks.rados.rados.0.smithi195.stdout:354: writing smithi19530025-22 from 1589248 to 2310144 tid 3 2024-09-16T02:15:28.019 INFO:tasks.rados.rados.0.smithi195.stdout:354: writing smithi19530025-22 from 2310144 to 2719744 tid 4 2024-09-16T02:15:28.020 INFO:tasks.rados.rados.0.smithi195.stdout:355: snap_create 2024-09-16T02:15:28.020 INFO:tasks.rados.rados.0.smithi195.stdout:351: expect (ObjNum 90 snap 12 seq_num 90) 2024-09-16T02:15:28.020 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:28.042 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1360064 2024-09-16T02:15:28.048 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 942967 2024-09-16T02:15:28.053 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 1 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 2 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 3 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 4 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 5 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: finishing write tid 6 to smithi19530025-2 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: oid 2 updating version 0 to 103 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: oid 2 version 103 is already newer than 102 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: oid 2 updating version 103 to 105 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: oid 2 version 105 is already newer than 104 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: oid 2 version 105 is already newer than 101 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 2 v 105 (ObjNum 137 snap 33 seq_num 137) dirty exists 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:350: left oid 2 (ObjNum 137 snap 33 seq_num 137) 2024-09-16T02:15:28.054 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 1 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 2 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 1 to smithi19530025-33 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 3 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 4 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 2 to smithi19530025-33 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 5 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 3 to smithi19530025-33 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 4 to smithi19530025-33 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: finishing write tid 6 to smithi19530025-10 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: oid 10 updating version 0 to 138 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: oid 10 version 138 is already newer than 137 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: oid 10 updating version 138 to 141 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: oid 10 version 141 is already newer than 140 2024-09-16T02:15:28.055 INFO:tasks.rados.rados.0.smithi195.stdout:352: oid 10 version 141 is already newer than 139 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 10 v 141 (ObjNum 138 snap 33 seq_num 138) dirty exists 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:352: left oid 10 (ObjNum 138 snap 33 seq_num 138) 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 5 to smithi19530025-33 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: finishing write tid 6 to smithi19530025-33 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: oid 33 updating version 0 to 74 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: oid 33 updating version 74 to 77 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: oid 33 version 77 is already newer than 73 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: oid 33 version 77 is already newer than 75 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: oid 33 version 77 is already newer than 76 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 33 v 77 (ObjNum 139 snap 33 seq_num 139) dirty exists 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:353: left oid 33 (ObjNum 139 snap 33 seq_num 139) 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 1 to smithi19530025-22 2024-09-16T02:15:28.056 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 2 to smithi19530025-22 2024-09-16T02:15:28.057 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 3 to smithi19530025-22 2024-09-16T02:15:28.057 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 4 to smithi19530025-22 2024-09-16T02:15:28.076 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:28.073+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:28.128 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.128 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.142 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:28.177 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:28.172+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:28.278 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:28.274+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:28.307 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.307 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:28.408 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:28.406+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:28.480 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:28.478+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:28.485 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.486 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.500 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:28.580 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:28.577+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:28.669 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.669 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.681 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:28.680+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:28.683 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:28.739 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 5 to smithi19530025-22 2024-09-16T02:15:28.741 INFO:tasks.rados.rados.0.smithi195.stdout:354: finishing write tid 6 to smithi19530025-22 2024-09-16T02:15:28.741 INFO:tasks.rados.rados.0.smithi195.stdout:354: oid 22 updating version 0 to 206 2024-09-16T02:15:28.741 INFO:tasks.rados.rados.0.smithi195.stdout:354: oid 22 updating version 206 to 209 2024-09-16T02:15:28.741 INFO:tasks.rados.rados.0.smithi195.stdout:354: oid 22 version 209 is already newer than 207 2024-09-16T02:15:28.741 INFO:tasks.rados.rados.0.smithi195.stdout:354: oid 22 version 209 is already newer than 208 2024-09-16T02:15:28.742 INFO:tasks.rados.rados.0.smithi195.stdout:354: oid 22 version 209 is already newer than 205 2024-09-16T02:15:28.742 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 22 v 209 (ObjNum 140 snap 33 seq_num 140) dirty exists 2024-09-16T02:15:28.742 INFO:tasks.rados.rados.0.smithi195.stdout:354: left oid 22 (ObjNum 140 snap 33 seq_num 140) 2024-09-16T02:15:28.782 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:28.782+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:28.804 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.804 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.818 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:28.883 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:28.880+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:350: done (5 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:351: done (4 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:352: done (3 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:353: done (2 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:354: done (1 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:355: done (0 left) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:356: read oid 4 snap 29 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:356: expect (ObjNum 125 snap 27 seq_num 125) 2024-09-16T02:15:28.930 INFO:tasks.rados.rados.0.smithi195.stdout:357: snap_create 2024-09-16T02:15:28.948 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:28.948 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:28.948 INFO:tasks.rados.rados.0.smithi195.stdout:356: expect (ObjNum 125 snap 27 seq_num 125) 2024-09-16T02:15:28.948 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:28.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:28.982 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:28.984 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:28.982+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:29.018 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 625338 2024-09-16T02:15:29.085 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:29.080+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:29.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:29.092 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:29.092 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:29.108 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:29.185 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:29.182+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:29.261 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:29.261 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:29.277 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:29.277 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:29.286 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:29.286+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:29.289 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:29.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:29.318 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:29.318 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:29.358 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:29.359 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:29.381 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:29.381 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:29.387 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:29.384+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:29.409 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:29.409 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:29.427 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:29.427 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:29.427 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:29.436 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:29.437 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:29.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:29.465 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:29.465 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:29.488 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:29.485+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:29.492 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:29.492 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:29.522 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:29.522 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:29.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:29.552 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:29.584 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:29.584 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:29.587 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:29.587 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:29.588 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:29.589 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:29.585+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:29.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:29.613 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:29.613 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:29.650 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:29.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:29.650+0000 7f60bf7fe640 0 -- 172.21.15.3:0/4112945683 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f60c021eb80 msgr2=0x7f60c021eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-16T02:15:29.690 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:29.686+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:76 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'54, mlcod=24'46)", 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:29.742 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.051862186999998, 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007242999, 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687518+0000", 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687520+0000", 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.796e-06 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687531+0000", 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1420000000000001e-05 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687607+0000", 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5166999999999994e-05 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.687620+0000", 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3873000000000001e-05 2024-09-16T02:15:29.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689759+0000", 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021384210000000002 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.689788+0000", 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8782999999999999e-05 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694680+0000", 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004892123 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.694761+0000", 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1415999999999994e-05 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:123 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'93, mlcod=24'87)", 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 82.817065764000006, 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069325319999999999, 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922315+0000", 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922316+0000", 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4640000000000001e-06 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.922381+0000", 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4969999999999996e-05 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924466+0000", 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002085075 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924479+0000", 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3033e-05 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924623+0000", 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014377899999999999 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924687+0000", 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3737000000000003e-05 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929168+0000", 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044810020000000004 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.929247+0000", 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9472000000000001e-05 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'89)", 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 82.814960053999997, 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33740482100000002, 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924422+0000", 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924431+0000", 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8769999999999999e-06 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260013+0000", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33558222100000001 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260031+0000", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8063e-05 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260228+0000", 2024-09-16T02:15:29.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000196782 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260273+0000", 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877000000000001e-05 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261745+0000", 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001471916 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261825+0000", 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9791999999999998e-05 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 45/32 ECSubWrite(tid=191, reqid=osd.13.0:191, at_version=45'28, trim_to=0'0, roll_forward_to=44'26))", 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.382375310999997, 2024-09-16T02:15:29.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071696049999999999, 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357005+0000", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357006+0000", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9399999999999995e-07 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357033+0000", 2024-09-16T02:15:29.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6978000000000002e-05 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357063+0000", 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0549999999999997e-05 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357074+0000", 2024-09-16T02:15:29.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1077000000000001e-05 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357740+0000", 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066576700000000005 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357753+0000", 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2706e-05 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364120+0000", 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063668939999999997 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364175+0000", 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5139000000000002e-05 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.379638393, 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068742350000000002, 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359742+0000", 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0699999999999998e-07 2024-09-16T02:15:29.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359769+0000", 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6655000000000001e-05 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359836+0000", 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7043000000000002e-05 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359848+0000", 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.199e-05 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359928+0000", 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9927999999999998e-05 2024-09-16T02:15:29.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359940+0000", 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.202e-05 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366543+0000", 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066029269999999998 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366616+0000", 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3264999999999997e-05 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 55.377302065999999, 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080847929999999998, 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362079+0000", 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.054e-06 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362106+0000", 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6322e-05 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362152+0000", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6276999999999998e-05 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362163+0000", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.064e-05 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362225+0000", 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2539999999999994e-05 2024-09-16T02:15:29.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362236+0000", 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1151999999999999e-05 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370090+0000", 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00785368 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370163+0000", 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3127999999999995e-05 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 66/32 ECSubWrite(tid=73, reqid=client.4598.0:580, at_version=66'64, trim_to=0'0, roll_forward_to=66'59))", 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 34.689865263000002, 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51414269899999998, 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.775 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049516+0000", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.68e-07 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049529+0000", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2897e-05 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562582+0000", 2024-09-16T02:15:29.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51305329300000002 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562593+0000", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1092e-05 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562676+0000", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2935000000000005e-05 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562686+0000", 2024-09-16T02:15:29.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0067e-05 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563613+0000", 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092661900000000005 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563658+0000", 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5127999999999999e-05 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=271, reqid=osd.13.0:271, at_version=73'80, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.787805162000002, 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071430060000000004, 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951575+0000", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951576+0000", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4700000000000001e-07 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951631+0000", 2024-09-16T02:15:29.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5352999999999997e-05 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951737+0000", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106183 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951760+0000", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2526000000000001e-05 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951944+0000", 2024-09-16T02:15:29.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000184112 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951971+0000", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6573000000000001e-05 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958634+0000", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066636960000000002 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.958718+0000", 2024-09-16T02:15:29.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3916000000000001e-05 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.786610249999999, 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073114410000000001, 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952772+0000", 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3300000000000001e-06 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952824+0000", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1836000000000003e-05 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952926+0000", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101223 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952948+0000", 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2478999999999999e-05 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953117+0000", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016898299999999999 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953144+0000", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7036e-05 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.959979+0000", 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068347410000000001 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.960082+0000", 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010281300000000001 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=274, reqid=osd.13.0:274, at_version=73'84, trim_to=0'0, roll_forward_to=73'82))", 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.425345658000001, 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047083613000000003, 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.788 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314037+0000", 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7940000000000001e-06 2024-09-16T02:15:29.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314080+0000", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3053000000000003e-05 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359609+0000", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045529636999999998 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359632+0000", 2024-09-16T02:15:29.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2869000000000001e-05 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359778+0000", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014631299999999999 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359798+0000", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9099000000000001e-05 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361025+0000", 2024-09-16T02:15:29.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012279420000000001 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361118+0000", 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2905999999999994e-05 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.780543042000001, 2024-09-16T02:15:29.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010173534, 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958839+0000", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7570000000000001e-06 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958892+0000", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2469999999999997e-05 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958988+0000", 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6779000000000003e-05 2024-09-16T02:15:29.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.959011+0000", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.245e-05 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961665+0000", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026538410000000001 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961690+0000", 2024-09-16T02:15:29.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5313e-05 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968914+0000", 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072245180000000001 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6405999999999998e-05 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.796 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.777919217000001, 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011431721000000001, 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961463+0000", 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.68e-06 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961515+0000", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2586000000000001e-05 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961613+0000", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7598999999999998e-05 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961635+0000", 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1780000000000002e-05 2024-09-16T02:15:29.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962881+0000", 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012456850000000001 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962897+0000", 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6303e-05 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972785+0000", 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098886259999999993 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972893+0000", 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107462 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.2s6 84/83 ECSubRead(tid=314, to_read={3:476b4430:::smithi19530025-30:14=262144,262144,0}, subchunks={3:476b4430:::smithi19530025-30:14=[0,1]}, attrs_to_read=))", 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:29.801 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.510308824999999, 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57667217800000004, 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229083+0000", 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0162e-05 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.804968+0000", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57588503999999996 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805000+0000", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2283000000000002e-05 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805176+0000", 2024-09-16T02:15:29.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017626499999999999 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805744+0000", 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056751299999999996 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=160, reqid=client.4598.0:758, at_version=85'151, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.656798835, 2024-09-16T02:15:29.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092037909999999994, 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:29.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000001e-07 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082613+0000", 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0790000000000002e-05 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082884+0000", 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027123099999999998 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082902+0000", 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.753e-05 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085344+0000", 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024427120000000001 2024-09-16T02:15:29.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085371+0000", 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6587000000000001e-05 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091675+0000", 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063039510000000003 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091785+0000", 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110436 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=161, reqid=client.4598.0:759, at_version=85'152, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.656763331000001, 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010507528, 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082618+0000", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082651+0000", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3246e-05 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083064+0000", 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041363000000000001 2024-09-16T02:15:29.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083085+0000", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.048e-05 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089109+0000", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060238330000000001 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089137+0000", 2024-09-16T02:15:29.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8569999999999999e-05 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093029+0000", 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038919520000000002 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093125+0000", 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5307000000000001e-05 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 85/83 ECSubWrite(tid=320, reqid=client.4598.0:756, at_version=85'108, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.647092167, 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076015120000000004, 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092289+0000", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9599999999999998e-07 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092436+0000", 2024-09-16T02:15:29.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000146983 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096144+0000", 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037081829999999999 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096168+0000", 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3782999999999999e-05 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096349+0000", 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018129600000000001 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096382+0000", 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2252000000000002e-05 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099809+0000", 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034276129999999999 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099890+0000", 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0605999999999994e-05 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 85/83 ECSubWrite(tid=321, reqid=osd.13.0:321, at_version=85'110, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.643416231, 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96998350899999997, 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095966+0000", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0540000000000002e-06 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096092+0000", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012574600000000001 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064646+0000", 2024-09-16T02:15:29.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96855418100000001 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064663+0000", 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.645e-05 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064759+0000", 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6565999999999997e-05 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064772+0000", 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2398000000000001e-05 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.065889+0000", 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011169979999999999 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.065948+0000", 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9116000000000001e-05 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.823 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 90/69 ECSubWrite(tid=173, reqid=client.4598.0:786, at_version=90'164, trim_to=0'0, roll_forward_to=90'161))", 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.742149947, 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107580065, 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997231+0000", 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9899999999999999e-07 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997248+0000", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6906e-05 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.102924+0000", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10567633799999999 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.102943+0000", 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8827000000000001e-05 2024-09-16T02:15:29.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.103113+0000", 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169727 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.103131+0000", 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7847e-05 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.104732+0000", 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016006919999999999 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.104811+0000", 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8929000000000003e-05 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 91/69 ECSubWrite(tid=178, reqid=osd.11.0:178, at_version=91'170, trim_to=0'0, roll_forward_to=90'164))", 2024-09-16T02:15:29.828 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 9.7747006259999996, 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007284876, 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964682+0000", 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2369999999999999e-06 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964741+0000", 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8904999999999999e-05 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964843+0000", 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010174 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964868+0000", 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5795999999999998e-05 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.965026+0000", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015726600000000001 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.965052+0000", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5930000000000001e-05 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.971919+0000", 2024-09-16T02:15:29.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006867153 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.971965+0000", 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5849000000000001e-05 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s4 91/83 ECSubWrite(tid=43, reqid=client.4598.0:838, at_version=91'58, trim_to=0'0, roll_forward_to=90'56))", 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:29.833 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 8.7932202020000005, 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083676029999999995, 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:29.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946161+0000", 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.113e-06 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948556+0000", 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023948060000000002 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:29.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948730+0000", 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000174105 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948768+0000", 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7423999999999999e-05 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948883+0000", 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011533599999999999 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948894+0000", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.136e-05 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.954390+0000", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005495977 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.954528+0000", 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137482 2024-09-16T02:15:29.837 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:29.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:29.838 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:29.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:29.840 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:29.838+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:29.889 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:29.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:29.922 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:29.922 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:29.931 INFO:tasks.rados.rados.0.smithi195.stdout:356: done (1 left) 2024-09-16T02:15:29.931 INFO:tasks.rados.rados.0.smithi195.stdout:357: done (0 left) 2024-09-16T02:15:29.931 INFO:tasks.rados.rados.0.smithi195.stdout:358: snap_create 2024-09-16T02:15:29.941 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:29.937+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:29.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:15:30.013 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:30.013 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:30.013 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:30.013 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:30.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:30.041 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:30.036+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:30.142 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:30.138+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:30.188 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:30.188 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:30.188 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:30.189 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:30.189 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:30.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:30.243 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:30.241+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:30.344 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:30.341+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:30.370 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=11, reqid=osd.12.0:11, at_version=45'6, trim_to=0'0, roll_forward_to=36'4))", 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.021313517000003, 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039104070000000003, 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347843+0000", 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347844+0000", 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9999999999999999e-07 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347874+0000", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0167000000000001e-05 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347952+0000", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8427e-05 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347965+0000", 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2937e-05 2024-09-16T02:15:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348056+0000", 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0390999999999996e-05 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.348066+0000", 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0584000000000001e-05 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351696+0000", 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003630084 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351754+0000", 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7516999999999998e-05 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.019535208000001, 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046999930000000004, 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349622+0000", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8400000000000001e-07 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349656+0000", 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4477999999999998e-05 2024-09-16T02:15:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349726+0000", 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9680999999999999e-05 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349739+0000", 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.323e-05 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349814+0000", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349826+0000", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1955e-05 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354271+0000", 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044454020000000002 2024-09-16T02:15:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.354322+0000", 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0506000000000001e-05 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=192, reqid=osd.13.0:192, at_version=45'18, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.011925603999998, 2024-09-16T02:15:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052360729999999999, 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357231+0000", 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357232+0000", 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.98e-07 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357254+0000", 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2855999999999999e-05 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5995000000000002e-05 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357322+0000", 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1798e-05 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357409+0000", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7282999999999998e-05 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357425+0000", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.52e-05 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362424+0000", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049990690000000001 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362467+0000", 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3674000000000002e-05 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.cs4 45/32 ECSubWrite(tid=8, reqid=osd.14.0:8, at_version=45'8, trim_to=0'0, roll_forward_to=36'6))", 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.011793461000003, 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065655959999999999, 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357363+0000", 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357364+0000", 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8199999999999999e-07 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357381+0000", 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7022999999999999e-05 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357419+0000", 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8779999999999998e-05 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357427+0000", 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357549+0000", 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121402 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.357561+0000", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2807e-05 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363880+0000", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063186550000000003 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.363929+0000", 2024-09-16T02:15:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9030999999999999e-05 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.9s3 45/32 ECSubWrite(tid=13, reqid=osd.12.0:11, at_version=0'0, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.009546311000001, 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045504229999999996, 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.508 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359611+0000", 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359612+0000", 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.122e-06 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359626+0000", 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4422e-05 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359678+0000", 2024-09-16T02:15:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1765999999999999e-05 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359690+0000", 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2143e-05 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359766+0000", 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5883000000000005e-05 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359776+0000", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0448999999999999e-05 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364111+0000", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043348809999999996 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.364161+0000", 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9756999999999998e-05 2024-09-16T02:15:30.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.3s4 45/32 ECSubWrite(tid=193, reqid=osd.13.0:193, at_version=45'20, trim_to=0'0, roll_forward_to=44'16))", 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 56.009251413000001, 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065736739999999998, 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359906+0000", 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:15:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359931+0000", 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4975999999999999e-05 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359972+0000", 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0244000000000003e-05 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.359982+0000", 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0710000000000001e-05 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360066+0000", 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3308999999999998e-05 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360080+0000", 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4363e-05 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366432+0000", 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00635214 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.366479+0000", 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6959999999999998e-05 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs2 52/32 ECSubWrite(tid=38, reqid=client.4598.0:394, at_version=52'23, trim_to=0'0, roll_forward_to=48'21))", 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.461305213000003, 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043643559999999998, 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907852+0000", 2024-09-16T02:15:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7e-07 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907890+0000", 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7839999999999997e-05 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.907996+0000", 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106345 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908022+0000", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5293999999999999e-05 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908228+0000", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206372 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.908273+0000", 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5225999999999997e-05 2024-09-16T02:15:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912158+0000", 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038845490000000002 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.912216+0000", 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8359999999999997e-05 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=54, reqid=client.4598.0:504, at_version=58'35, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.356970472, 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048877479999999999, 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012186+0000", 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.012189+0000", 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0559999999999999e-06 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013086+0000", 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089770099999999999 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013357+0000", 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027122399999999998 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013374+0000", 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6160000000000001e-05 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013559+0000", 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185515 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013585+0000", 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6037000000000002e-05 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016983+0000", 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033975289999999998 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017074+0000", 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1526000000000004e-05 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=55, reqid=client.4598.0:505, at_version=58'36, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.356062182000002, 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042762490000000002, 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013095+0000", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1399999999999997e-07 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013218+0000", 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012275200000000001 2024-09-16T02:15:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013417+0000", 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019934400000000001 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013427+0000", 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3370000000000006e-06 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015391+0000", 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019645779999999998 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015399+0000", 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017304+0000", 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001904953 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017371+0000", 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6846000000000003e-05 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=56, reqid=client.4598.0:506, at_version=58'37, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.355906152999999, 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056867990000000002, 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013251+0000", 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8999999999999997e-07 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013335+0000", 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4120000000000001e-05 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013545+0000", 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020994500000000001 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013558+0000", 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3044000000000001e-05 2024-09-16T02:15:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.015997+0000", 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024385710000000001 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.016025+0000", 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7932999999999998e-05 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018839+0000", 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028140270000000002 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.018938+0000", 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8669e-05 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 58/32 ECSubWrite(tid=57, reqid=client.4598.0:507, at_version=58'38, trim_to=0'0, roll_forward_to=54'33))", 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.355548853000002, 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059371409999999999, 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013608+0000", 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013609+0000", 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7999999999999995e-07 2024-09-16T02:15:30.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013650+0000", 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1192000000000001e-05 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013688+0000", 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7902999999999999e-05 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.013698+0000", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9690000000000006e-06 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017129+0000", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034309319999999998 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.017136+0000", 2024-09-16T02:15:30.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.396e-06 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019482+0000", 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002345813 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:47.019545+0000", 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3355999999999994e-05 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.542 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 73/32 ECSubWrite(tid=104, reqid=osd.5.0:104, at_version=73'71, trim_to=0'0, roll_forward_to=72'69))", 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.426218935000001, 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051077429999999997, 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.942938+0000", 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3700000000000001e-07 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943033+0000", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4955000000000001e-05 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943150+0000", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116166 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943181+0000", 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1717000000000001e-05 2024-09-16T02:15:30.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943369+0000", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018737199999999999 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.943404+0000", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5102999999999999e-05 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.947951+0000", 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045476809999999996 2024-09-16T02:15:30.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.948046+0000", 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4312000000000001e-05 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 73/32 ECSubWrite(tid=105, reqid=osd.5.0:105, at_version=73'73, trim_to=0'0, roll_forward_to=72'69))", 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.425046623, 2024-09-16T02:15:30.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078423599999999996, 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944110+0000", 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944112+0000", 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7400000000000001e-06 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944179+0000", 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6805999999999997e-05 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944269+0000", 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0508999999999997e-05 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.944293+0000", 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.393e-05 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.945156+0000", 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00086232699999999997 2024-09-16T02:15:30.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.945183+0000", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7327000000000001e-05 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951860+0000", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066769079999999996 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.951953+0000", 2024-09-16T02:15:30.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2812999999999994e-05 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.413971367999999, 2024-09-16T02:15:30.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038733359999999998, 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955186+0000", 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.37e-07 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955246+0000", 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0217000000000003e-05 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955369+0000", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012299199999999999 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955393+0000", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3589999999999999e-05 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955564+0000", 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170858 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.955596+0000", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2539999999999997e-05 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958958+0000", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033615889999999999 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.959059+0000", 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100913 2024-09-16T02:15:30.556 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.411490114999999, 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087020870000000007, 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957667+0000", 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957669+0000", 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.72e-06 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957733+0000", 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4912999999999993e-05 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957814+0000", 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0241999999999995e-05 2024-09-16T02:15:30.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957829+0000", 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5007e-05 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.957982+0000", 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153215 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958008+0000", 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5927000000000001e-05 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.966293+0000", 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082847939999999998 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.966369+0000", 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6268999999999998e-05 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.40755721, 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075408059999999997, 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961600+0000", 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961602+0000", 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.863e-06 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961656+0000", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4599999999999999e-05 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961776+0000", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119926 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961802+0000", 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5709999999999999e-05 2024-09-16T02:15:30.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961956+0000", 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015436200000000001 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961983+0000", 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6460000000000001e-05 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969059+0000", 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070767859999999998 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969141+0000", 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1099000000000004e-05 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s3 74/32 ECSubWrite(tid=113, reqid=client.4598.0:656, at_version=74'83, trim_to=0'0, roll_forward_to=74'77))", 2024-09-16T02:15:30.566 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.395822903999999, 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035945349999999998, 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973334+0000", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973336+0000", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9999999999999999e-06 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973380+0000", 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4026000000000002e-05 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973485+0000", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104776 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973510+0000", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5333e-05 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973675+0000", 2024-09-16T02:15:30.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016454600000000001 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.973701+0000", 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6676999999999999e-05 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.976856+0000", 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003154858 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.976929+0000", 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2318999999999997e-05 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s3 85/83 ECSubWrite(tid=320, reqid=client.4598.0:756, at_version=85'108, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.278416348, 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005788835, 2024-09-16T02:15:30.571 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090741+0000", 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090743+0000", 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2199999999999999e-06 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091068+0000", 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000325414 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091179+0000", 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110759 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091204+0000", 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5227000000000001e-05 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091508+0000", 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030419099999999999 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091551+0000", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2215999999999997e-05 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096441+0000", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048900719999999996 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096529+0000", 2024-09-16T02:15:30.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8735999999999999e-05 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s3 91/83 ECSubWrite(tid=323, reqid=client.4598.0:824, at_version=91'112, trim_to=0'0, roll_forward_to=85'110))", 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.993638+0000", 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.375519391999999, 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042497489999999997, 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.576 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993638+0000", 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993638+0000", 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993638+0000", 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.37e-07 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993822+0000", 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018386800000000001 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993906+0000", 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3555999999999998e-05 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993932+0000", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6023000000000001e-05 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.994090+0000", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015831800000000001 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.994112+0000", 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2252e-05 2024-09-16T02:15:30.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997791+0000", 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036786060000000001 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997887+0000", 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6589000000000006e-05 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s3 95/83 ECSubWrite(tid=336, reqid=client.4598.0:853, at_version=95'123, trim_to=0'0, roll_forward_to=93'121))", 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:24.006134+0000", 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 6.3630225840000003, 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010243054, 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.006134+0000", 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.006134+0000", 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:30.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.006135+0000", 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8500000000000002e-07 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.008371+0000", 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022363510000000001 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.008419+0000", 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7808e-05 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.008433+0000", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4199000000000001e-05 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.008540+0000", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106316 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.008553+0000", 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3920000000000001e-05 2024-09-16T02:15:30.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.016305+0000", 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077513299999999999 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:24.016377+0000", 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2645000000000007e-05 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:30.586 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:30.587 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:30.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:30.589 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:30.586+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:30.690 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:30.684+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:30.723 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:30.724 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:30.724 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:30.724 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:30.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:30.791 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:30.789+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:30.866 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:30.867 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:30.867 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:30.867 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:30.867 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:30.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:30.926 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:30.926+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:30.927 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:15:30.933 INFO:tasks.rados.rados.0.smithi195.stdout:358: done (0 left) 2024-09-16T02:15:30.933 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 7 current snap is 36 2024-09-16T02:15:30.933 INFO:tasks.rados.rados.0.smithi195.stdout:359: seq_num 141 ranges {344064=425984,770048=573440,1343488=425984,1769472=524288,2293760=425984} 2024-09-16T02:15:30.941 INFO:tasks.rados.rados.0.smithi195.stdout:359: writing smithi19530025-7 from 344064 to 770048 tid 1 2024-09-16T02:15:30.947 INFO:tasks.rados.rados.0.smithi195.stdout:359: writing smithi19530025-7 from 770048 to 1343488 tid 2 2024-09-16T02:15:30.952 INFO:tasks.rados.rados.0.smithi195.stdout:359: writing smithi19530025-7 from 1343488 to 1769472 tid 3 2024-09-16T02:15:30.957 INFO:tasks.rados.rados.0.smithi195.stdout:359: writing smithi19530025-7 from 1769472 to 2293760 tid 4 2024-09-16T02:15:30.962 INFO:tasks.rados.rados.0.smithi195.stdout:359: writing smithi19530025-7 from 2293760 to 2719744 tid 5 2024-09-16T02:15:30.962 INFO:tasks.rados.rados.0.smithi195.stdout:360: delete oid 14 current snap is 36 2024-09-16T02:15:30.962 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 1 to smithi19530025-7 2024-09-16T02:15:30.962 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 2 to smithi19530025-7 2024-09-16T02:15:30.968 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 3 to smithi19530025-7 2024-09-16T02:15:30.973 INFO:tasks.rados.rados.0.smithi195.stdout:360: done (1 left) 2024-09-16T02:15:30.973 INFO:tasks.rados.rados.0.smithi195.stdout:361: snap_remove snap 32 2024-09-16T02:15:30.993 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:30.990+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:489 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:31.013 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 45.078359374999998, 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092923889999999999, 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 489 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932945+0000", 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932946+0000", 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932970+0000", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932983+0000", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.27e-05 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.932991+0000", 2024-09-16T02:15:31.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999999e-06 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933148+0000", 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156828 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933157+0000", 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8409999999999995e-06 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938656+0000", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005498541 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.938725+0000", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9061999999999997e-05 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.942237+0000", 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035124739999999998 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.018 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:490 3.es0 3:701f1021:::smithi19530025-33:head [read 0~540672,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 45.078275464999997, 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010928416999999999, 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 490 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933029+0000", 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933030+0000", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6700000000000003e-07 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933034+0000", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7169999999999999e-06 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933046+0000", 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1584e-05 2024-09-16T02:15:31.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933060+0000", 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3953e-05 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933191+0000", 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131347 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:31.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.933199+0000", 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7030000000000006e-06 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939244+0000", 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060455639999999998 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.939311+0000", 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6313e-05 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:45.943957+0000", 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046465689999999997 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:492 3.es0 3:701f1021:::smithi19530025-33:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 44.061506813999998, 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017209518, 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.024 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 492 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949798+0000", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949801+0000", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2100000000000002e-06 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949954+0000", 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015366300000000001 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.949991+0000", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6463e-05 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950016+0000", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4593e-05 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950156+0000", 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000140882 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.950172+0000", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5590000000000002e-05 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954336+0000", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041644309999999997 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.954560+0000", 2024-09-16T02:15:31.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022320200000000001 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.956344+0000", 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017841000000000001 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.958842+0000", 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024979220000000001 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.966762+0000", 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079201440000000005 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:46.967007+0000", 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024531800000000001 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:653 3.es0 3:701f1021:::smithi19530025-33:head [rmxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ in=40b,rmxattr OOq^bpU^bm in=10b,rmxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX in=40b,rmxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ in=40b,rmxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU in=40b,rmxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe in=40b,rmxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn in=40b,rmxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr in=40b,rmxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ in=40b,rmxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a in=40b,rmxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp in=40b,rmxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV in=40b,rmxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS in=40b,rmxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR in=40b,rmxattr biYdnofqNd_eZg\\amb``Na_saZ^ in=27b,rmxattr fceY\\cr[bnaj[ in=13b,rmxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq in=40b,rmxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t in=40b,rmxattr olt]ab in=6b,rmxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf in=40b] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:31.030 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.054915792999999, 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025463508999999999, 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 653 2024-09-16T02:15:31.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956389+0000", 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956393+0000", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.454e-06 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956416+0000", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2685999999999999e-05 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956463+0000", 2024-09-16T02:15:31.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7349999999999999e-05 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956488+0000", 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4983000000000001e-05 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956673+0000", 2024-09-16T02:15:31.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000184895 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.956698+0000", 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4905999999999998e-05 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969407+0000", 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012709420000000001 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969740+0000", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033311700000000002 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972472+0000", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027318680000000001 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.978868+0000", 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063953710000000004 2024-09-16T02:15:31.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.981617+0000", 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027492340000000001 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.981852+0000", 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023522500000000001 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.es0 74/66 ECSubWriteReply(tid=69, last_complete=74'46, committed=1, applied=1))", 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.032192381000002, 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.58875802499999996, 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979112+0000", 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979114+0000", 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.643e-06 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.979125+0000", 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1399999999999999e-05 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566505+0000", 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.58737945199999997 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566518+0000", 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.385e-05 2024-09-16T02:15:31.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.566681+0000", 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000162313 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.567870+0000", 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001189367 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.7s1 74/32 ECSubWrite(tid=94, reqid=osd.3.0:94, at_version=74'91, trim_to=0'0, roll_forward_to=73'87))", 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 28.023196468999998, 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.45024945, 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:31.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988108+0000", 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988110+0000", 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6729999999999999e-06 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.043 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.988147+0000", 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7766999999999999e-05 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436677+0000", 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.44852915199999999 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436695+0000", 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8009999999999999e-05 2024-09-16T02:15:31.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436859+0000", 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016417499999999999 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.436880+0000", 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0933000000000001e-05 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.045 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.438276+0000", 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013959269999999999 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:03.438357+0000", 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1812999999999997e-05 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:718 3.es0 3:7ab7d2d0:::smithi19530025-50:d [read 0~2031616,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:31.046 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 22.978462381, 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092095790000000007, 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 718 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:31.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032842+0000", 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3599999999999999e-07 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032845+0000", 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9399999999999998e-06 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032856+0000", 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0881e-05 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.032861+0000", 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5700000000000003e-06 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033868+0000", 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00100712 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.033940+0000", 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1804000000000006e-05 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:08.042052+0000", 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081119279999999992 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpReadReply(3.es0 84/83 ECSubReadReply(tid=86, attrs_read=0))", 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.870028264999998, 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.77097236800000002, 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141276+0000", 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141277+0000", 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6700000000000002e-07 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.141534+0000", 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025689899999999999 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911063+0000", 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.76952898400000003 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911077+0000", 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4446e-05 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.911215+0000", 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013733300000000001 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.912249+0000", 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010338389999999999 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 84/83 [PushReplyOp(3:7df442af:::smithi19530025-39:14)])", 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.09680238, 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17568129900000001, 2024-09-16T02:15:31.055 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914502+0000", 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914504+0000", 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9599999999999999e-06 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.914518+0000", 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4306e-05 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089101+0000", 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17458259700000001 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089122+0000", 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1399999999999998e-05 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089292+0000", 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016961600000000001 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.090183+0000", 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089141999999999997 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 85/83 [PushReplyOp(3:7df442af:::smithi19530025-39:head)])", 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.922268269, 2024-09-16T02:15:31.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012806684, 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089036+0000", 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089038+0000", 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6360000000000001e-06 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089051+0000", 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3499999999999999e-05 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101196+0000", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012145100000000001 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101215+0000", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8496999999999999e-05 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101344+0000", 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128835 2024-09-16T02:15:31.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.101843+0000", 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049911599999999997 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(3.es0 85/83 [PushReplyOp(3:7df442af:::smithi19530025-39:head)])", 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 16.903945054000001, 2024-09-16T02:15:31.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42809435099999998, 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107359+0000", 2024-09-16T02:15:31.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107361+0000", 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.784e-06 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.107372+0000", 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.045e-05 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535025+0000", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.42765297299999999 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535042+0000", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7767000000000001e-05 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535170+0000", 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128119 2024-09-16T02:15:31.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.535454+0000", 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028325799999999999 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:767 3.es0 3:7df442af:::smithi19530025-39:head [copy-from ver 49] snapc 1a=[1a,18,d] ondisk+write+known_if_redirected+supports_pool_eio e86)", 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.936966427, 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026046024000000001, 2024-09-16T02:15:31.067 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 767 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074338+0000", 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074340+0000", 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3700000000000002e-06 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074350+0000", 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7650000000000005e-06 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074371+0000", 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0914999999999999e-05 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074380+0000", 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1060000000000006e-06 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074459+0000", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8447000000000003e-05 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074466+0000", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2439999999999996e-06 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076762+0000", 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022956880000000002 2024-09-16T02:15:31.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076829+0000", 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7267999999999994e-05 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.093614+0000", 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016785255999999998 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.096089+0000", 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024751349999999998 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.100271+0000", 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041815380000000003 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.100384+0000", 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113292 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.073 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:768 3.es0 3:7df442af:::smithi19530025-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e86)", 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 15.936905903, 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021202618999999999, 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 768 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:31.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074399+0000", 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0499999999999999e-07 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074401+0000", 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9379999999999999e-06 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074416+0000", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4816e-05 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074422+0000", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4420000000000003e-06 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074523+0000", 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100993 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.074529+0000", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2899999999999999e-06 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.076959+0000", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024291550000000001 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.077007+0000", 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8551000000000003e-05 2024-09-16T02:15:31.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095460+0000", 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.018452402 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095525+0000", 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5692999999999996e-05 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.095601+0000", 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5933999999999995e-05 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:818 3.es0 3:7ab7d2d0:::smithi19530025-50:head [append 0~589824] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.981389+0000", 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.029915461, 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096679490000000003, 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.080 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 818 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.981389+0000", 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.981389+0000", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.981392+0000", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6929999999999999e-06 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.982459+0000", 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001066956 2024-09-16T02:15:31.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.982493+0000", 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4449000000000002e-05 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.982509+0000", 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5444e-05 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.982620+0000", 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011134299999999999 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.982761+0000", 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014163700000000001 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.984351+0000", 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001589561 2024-09-16T02:15:31.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990657+0000", 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063057240000000004 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990915+0000", 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025819100000000001 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991057+0000", 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141951 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:819 3.es0 3:7ab7d2d0:::smithi19530025-50:head [append 0~753664] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.986086+0000", 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.025218065000001, 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011326925, 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.086 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 819 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.986086+0000", 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.986086+0000", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.986089+0000", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3499999999999999e-06 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.987272+0000", 2024-09-16T02:15:31.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011836220000000001 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.987305+0000", 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2523e-05 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.987323+0000", 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8442999999999999e-05 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.987421+0000", 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7989e-05 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.987605+0000", 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000183291 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.989072+0000", 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001466946 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.993905+0000", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004833337 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997326+0000", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034211829999999999 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997413+0000", 2024-09-16T02:15:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7241e-05 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:820 3.es0 3:7ab7d2d0:::smithi19530025-50:head [append 0~475136] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.988961+0000", 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.022343545, 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012192556, 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.092 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 820 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.988961+0000", 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.093 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.988961+0000", 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.988964+0000", 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2600000000000001e-06 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990144+0000", 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001180031 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990179+0000", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4736000000000001e-05 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990197+0000", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8147000000000001e-05 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991144+0000", 2024-09-16T02:15:31.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094678999999999998 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991320+0000", 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017622800000000001 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.992438+0000", 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011180820000000001 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.996912+0000", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004473861 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.001092+0000", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041796289999999998 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.001153+0000", 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1791999999999994e-05 2024-09-16T02:15:31.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:821 3.es0 3:7ab7d2d0:::smithi19530025-50:head [append 0~327680] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.990944+0000", 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.020360749, 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011105762999999999, 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.098 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 821 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990944+0000", 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990944+0000", 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.990946+0000", 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4360000000000001e-06 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991687+0000", 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074053499999999996 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991719+0000", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2255999999999997e-05 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991739+0000", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0132999999999998e-05 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.994235+0000", 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024964079999999999 2024-09-16T02:15:31.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.994452+0000", 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000216443 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.996012+0000", 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015596620000000001 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.999375+0000", 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033637390000000001 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.001868+0000", 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024927460000000001 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.002049+0000", 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018140499999999999 2024-09-16T02:15:31.103 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:822 3.es0 3:7ab7d2d0:::smithi19530025-50:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.991774+0000", 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.019530389, 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011828103, 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.104 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 822 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991774+0000", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991774+0000", 2024-09-16T02:15:31.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991776+0000", 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991785+0000", 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2269999999999999e-06 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991806+0000", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0869999999999998e-05 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991822+0000", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6256000000000002e-05 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997646+0000", 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058243690000000002 2024-09-16T02:15:31.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.997822+0000", 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017583799999999999 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.998677+0000", 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085503399999999996 2024-09-16T02:15:31.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.000365+0000", 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016882010000000001 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.003492+0000", 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031261090000000002 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.003602+0000", 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110559 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:823 3.es0 3:7ab7d2d0:::smithi19530025-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.991851+0000", 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 11.019453903, 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013605826, 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.110 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 823 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991851+0000", 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991851+0000", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991852+0000", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1799999999999999e-06 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991859+0000", 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9469999999999997e-06 2024-09-16T02:15:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991880+0000", 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1736e-05 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.991900+0000", 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9737000000000001e-05 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.999426+0000", 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075260530000000004 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.999498+0000", 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1834000000000004e-05 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.004017+0000", 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045191930000000003 2024-09-16T02:15:31.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.004073+0000", 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6155e-05 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.005456+0000", 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013829910000000001 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s6 91/83 ECSubWrite(tid=43, reqid=client.4598.0:838, at_version=91'58, trim_to=0'0, roll_forward_to=90'56))", 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:20.946591+0000", 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.064713424000001, 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091059910000000008, 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946591+0000", 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946591+0000", 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946592+0000", 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4600000000000004e-07 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.947499+0000", 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00090719399999999995 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.947625+0000", 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012652200000000001 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.947697+0000", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1649999999999993e-05 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948223+0000", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052629 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948252+0000", 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9119e-05 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.955591+0000", 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073384009999999996 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.955697+0000", 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010616900000000001 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.121 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.121 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.121 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:31.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:31.123 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:31.120+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:31.223 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:31.216+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:31.254 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:31.254 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:31.254 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:31.254 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:31.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:31.323 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:31.322+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:31.395 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:31.395 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:31.395 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:31.395 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:31.396 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:31.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:31.424 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:31.421+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:31.525 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:31.521+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:46 3.0s0 3:0952f51a:::smithi19530025-16:head [append 0~262144,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.282219890999997, 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013865147, 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300539+0000", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.300541+0000", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.193e-06 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301465+0000", 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092351300000000005 2024-09-16T02:15:31.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301478+0000", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.348e-05 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301487+0000", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3869999999999995e-06 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301559+0000", 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2444e-05 2024-09-16T02:15:31.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301744+0000", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000185019 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.302843+0000", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010987550000000001 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305418+0000", 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025746219999999999 2024-09-16T02:15:31.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314319+0000", 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089012259999999999 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314404+0000", 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5507999999999995e-05 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:47 3.0s0 3:0952f51a:::smithi19530025-16:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.281251631000003, 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013156407, 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 47 2024-09-16T02:15:31.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301507+0000", 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301508+0000", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.079e-06 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301513+0000", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.194e-06 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301521+0000", 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2309999999999996e-06 2024-09-16T02:15:31.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301527+0000", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4710000000000004e-06 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304040+0000", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025124589999999999 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304279+0000", 2024-09-16T02:15:31.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023908499999999999 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.304869+0000", 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058981500000000004 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.307149+0000", 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022805849999999999 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314589+0000", 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074399310000000003 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314664+0000", 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4556999999999998e-05 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:48 3.0s0 3:0952f51a:::smithi19530025-16:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 67.281065847999997, 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.021785207000000001, 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 48 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.598 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301693+0000", 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0359999999999999e-06 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301701+0000", 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9700000000000002e-06 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301718+0000", 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7201999999999999e-05 2024-09-16T02:15:31.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.301729+0000", 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0516999999999999e-05 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305321+0000", 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035922570000000002 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.305405+0000", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3998000000000005e-05 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.314939+0000", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095341579999999992 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.315042+0000", 2024-09-16T02:15:31.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102858 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:24.323478+0000", 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0084362110000000008 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:252 3.0s0 3:0952f51a:::smithi19530025-16:head [stat,delete] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:31.603 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 57.237831993, 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011932368, 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 252 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344927+0000", 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344929+0000", 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2709999999999999e-06 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344944+0000", 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.469e-05 2024-09-16T02:15:31.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344959+0000", 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4990999999999999e-05 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.344967+0000", 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9670000000000001e-06 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345036+0000", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9412999999999994e-05 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.345046+0000", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6390000000000004e-06 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346669+0000", 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016225289999999999 2024-09-16T02:15:31.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.346810+0000", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141457 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.347909+0000", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010991670000000001 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.349212+0000", 2024-09-16T02:15:31.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013032880000000001 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356764+0000", 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007551978 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4977999999999998e-05 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.fs1 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 57.231592075000002, 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095540220000000006, 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351167+0000", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351168+0000", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7899999999999997e-07 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351191+0000", 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3377e-05 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351238+0000", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351247+0000", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2310000000000002e-06 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351304+0000", 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7058000000000002e-05 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.351312+0000", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8790000000000003e-06 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360676+0000", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0093640279999999999 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360721+0000", 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4975999999999998e-05 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.615 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWriteReply(3.0s0 45/32 ECSubWriteReply(tid=12, last_complete=45'3, committed=1, applied=1))", 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 57.221806729000001, 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82653599300000002, 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:31.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360952+0000", 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360954+0000", 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42e-06 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.360960+0000", 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6340000000000003e-06 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187311+0000", 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.82635024400000001 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187328+0000", 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7880000000000002e-05 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187449+0000", 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012031000000000001 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:35.187488+0000", 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9505000000000002e-05 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:404 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~606208] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:31.619 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.620574425000001, 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022657539000000001, 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 404 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:31.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962185+0000", 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.962187+0000", 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8729999999999999e-06 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963628+0000", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001440113 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963678+0000", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0349999999999997e-05 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963702+0000", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4533e-05 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.963873+0000", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170669 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.964084+0000", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021121200000000001 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.966544+0000", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024596430000000001 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.969112+0000", 2024-09-16T02:15:31.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025675950000000002 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984664+0000", 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015552145999999999 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984842+0000", 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178405 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:405 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~720896] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.608259998999998, 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011549382, 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 405 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974499+0000", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.974502+0000", 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4959999999999999e-06 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975490+0000", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00098815699999999992 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975523+0000", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3583e-05 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975539+0000", 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5330000000000001e-05 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975692+0000", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015385600000000001 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975935+0000", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024296100000000001 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.977706+0000", 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017700909999999999 2024-09-16T02:15:31.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984097+0000", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063910879999999996 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985929+0000", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018325940000000001 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986048+0000", 2024-09-16T02:15:31.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119226 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:406 3.9s0 3:9a381d46:::smithi19530025-47:head [append 0~49152] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.607186560000002, 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011213219999999999, 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 406 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975572+0000", 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975574+0000", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4300000000000001e-06 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975631+0000", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7519999999999998e-05 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975655+0000", 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3465999999999999e-05 2024-09-16T02:15:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975671+0000", 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6186999999999999e-05 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.979950+0000", 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042792400000000001 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.980113+0000", 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016220199999999999 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.981185+0000", 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001072092 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.984325+0000", 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031403780000000001 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986680+0000", 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023551359999999999 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986786+0000", 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010556900000000001 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:407 3.9s0 3:9a381d46:::smithi19530025-47:head [setxattr _header (57) in=7b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:31.635 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.607058260999999, 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.35746389899999997, 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 407 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975701+0000", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975702+0000", 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.57e-06 2024-09-16T02:15:31.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975711+0000", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2900000000000002e-06 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975730+0000", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9017000000000001e-05 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975749+0000", 2024-09-16T02:15:31.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8850000000000001e-05 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982236+0000", 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064873109999999999 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.982511+0000", 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027516800000000001 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.983460+0000", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00094918300000000004 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.985743+0000", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00228254 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.332957+0000", 2024-09-16T02:15:31.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.347213828 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333165+0000", 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000208142 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:408 3.9s0 3:9a381d46:::smithi19530025-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:31.641 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 50.606988344999998, 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.36004730099999999, 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 408 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975771+0000", 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975772+0000", 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.544e-06 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975779+0000", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2829999999999998e-06 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975801+0000", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2082999999999998e-05 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.975816+0000", 2024-09-16T02:15:31.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5662999999999999e-05 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.986954+0000", 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011137461 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:40.987093+0000", 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139436 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.333971+0000", 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.34687802400000001 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.334106+0000", 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013453200000000001 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:41.335818+0000", 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017122750000000001 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.4s1 85/66 ECSubWrite(tid=171, reqid=client.4598.0:760, at_version=85'57, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.080315+0000", 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.502444475000001, 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011726769999999999, 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.080315+0000", 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.080315+0000", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.080315+0000", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000005e-07 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.080623+0000", 2024-09-16T02:15:31.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030780800000000002 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.087630+0000", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070070769999999996 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.087655+0000", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4726e-05 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.087799+0000", 2024-09-16T02:15:31.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014401299999999999 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.087826+0000", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6709000000000001e-05 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091953+0000", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041272970000000003 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092041+0000", 2024-09-16T02:15:31.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8453000000000002e-05 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.4s1 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.093180+0000", 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.489579057, 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.231811235, 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093180+0000", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093180+0000", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093182+0000", 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8640000000000001e-06 2024-09-16T02:15:31.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093224+0000", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2052999999999999e-05 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.323771+0000", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23054676299999999 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.323785+0000", 2024-09-16T02:15:31.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4453999999999999e-05 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.323871+0000", 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6044000000000005e-05 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.323881+0000", 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.028e-05 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.324897+0000", 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010157180000000001 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.324991+0000", 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4059000000000002e-05 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:806 3.5s0 3:aae22120:::smithi19530025-4:head [append 0~507904] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.034261+0000", 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.548497811000001, 2024-09-16T02:15:31.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013689488, 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 806 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034261+0000", 2024-09-16T02:15:31.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034261+0000", 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034263+0000", 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1220000000000002e-06 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034938+0000", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00067445099999999996 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034959+0000", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0755999999999999e-05 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.034972+0000", 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2958e-05 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.035685+0000", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00071387299999999996 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.035810+0000", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124213 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.036772+0000", 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00096253500000000004 2024-09-16T02:15:31.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.047776+0000", 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011004207 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.047852+0000", 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5921999999999996e-05 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.047951+0000", 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8450999999999996e-05 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:807 3.5s0 3:aae22120:::smithi19530025-4:head [append 0~475136] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.037459+0000", 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.545300065999999, 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010828295999999999, 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 807 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.037459+0000", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.037459+0000", 2024-09-16T02:15:31.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.037462+0000", 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8040000000000002e-06 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.037943+0000", 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048078200000000002 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.037982+0000", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9539999999999998e-05 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038075+0000", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2578999999999996e-05 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038790+0000", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00071510900000000001 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.039064+0000", 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027450800000000002 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.040462+0000", 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001397216 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.048136+0000", 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076740829999999999 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.048196+0000", 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0590000000000001e-05 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.048287+0000", 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1085e-05 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:808 3.5s0 3:aae22120:::smithi19530025-4:head [append 0~98304] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:31.666 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.038191+0000", 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.544567978, 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011517248000000001, 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 808 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038191+0000", 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038191+0000", 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038193+0000", 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0899999999999999e-06 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038485+0000", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029207799999999998 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038516+0000", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1046000000000001e-05 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038536+0000", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9233999999999999e-05 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.043662+0000", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051262629999999998 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.043815+0000", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015353200000000001 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.044783+0000", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00096745299999999995 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.048835+0000", 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040526520000000003 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.049530+0000", 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069484400000000002 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.049708+0000", 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178056 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:809 3.5s0 3:aae22120:::smithi19530025-4:head [setxattr _header (56) in=7b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.038567+0000", 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.54419188, 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012315888000000001, 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 809 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.672 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038567+0000", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038567+0000", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038569+0000", 2024-09-16T02:15:31.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.626e-06 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038579+0000", 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0497e-05 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038599+0000", 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9953e-05 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038614+0000", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4752999999999999e-05 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.045841+0000", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072269420000000001 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.046091+0000", 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024967499999999998 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.046979+0000", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088796299999999997 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.049931+0000", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029525689999999999 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.050040+0000", 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010933999999999999 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.050883+0000", 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084256999999999995 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:810 3.5s0 3:aae22120:::smithi19530025-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.038640+0000", 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 12.544119005000001, 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015318330999999999, 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 810 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038640+0000", 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038640+0000", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038642+0000", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6670000000000001e-06 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038648+0000", 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5400000000000001e-06 2024-09-16T02:15:31.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038668+0000", 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9813000000000002e-05 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.038684+0000", 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6200000000000001e-05 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.049097+0000", 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010412988 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.049216+0000", 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119021 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.050960+0000", 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001744099 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.051066+0000", 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010578600000000001 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.053958+0000", 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028922169999999999 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.682 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:838 3.5s0 3:a2a1f4c4:::smithi19530025-25:head [copy-from ver 64] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:20.913014+0000", 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.669745077, 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.043312195999999997, 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 838 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913014+0000", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913014+0000", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913018+0000", 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0929999999999996e-06 2024-09-16T02:15:31.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913036+0000", 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7640000000000001e-05 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913074+0000", 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8245999999999999e-05 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913100+0000", 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6359999999999998e-05 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913201+0000", 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100509 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913417+0000", 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021569100000000001 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946688+0000", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033271843000000002 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.955750+0000", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.009062011 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.956156+0000", 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040546400000000002 2024-09-16T02:15:31.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.956326+0000", 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017033900000000001 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.4598.0:839 3.5s0 3:a2a1f4c4:::smithi19530025-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e91)", 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:20.913151+0000", 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 10.669607778, 2024-09-16T02:15:31.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042832949000000002, 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.4598", 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 839 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913151+0000", 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913151+0000", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913153+0000", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.57e-06 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913160+0000", 2024-09-16T02:15:31.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4560000000000003e-06 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913183+0000", 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2387000000000001e-05 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913201+0000", 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8785999999999999e-05 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913647+0000", 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044510399999999999 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.913717+0000", 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0151e-05 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.953903+0000", 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.040185787000000001 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.954142+0000", 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023964799999999999 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.955984+0000", 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00184206 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:31.693 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:31.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:31.695 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:31.692+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:31.796 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:31.794+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:31.828 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:31.828 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:31.828 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:31.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:31.896 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:31.892+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:361: done (1 left) 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:362: copy_from oid 37 from oid 23 current snap is 36 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 4 to smithi19530025-7 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 5 to smithi19530025-7 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 6 to smithi19530025-7 2024-09-16T02:15:31.937 INFO:tasks.rados.rados.0.smithi195.stdout:359: finishing write tid 7 to smithi19530025-7 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 updating version 0 to 62 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 version 62 is already newer than 61 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 version 62 is already newer than 60 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 updating version 62 to 64 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 version 64 is already newer than 63 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: oid 7 updating version 64 to 65 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 7 v 65 (ObjNum 141 snap 36 seq_num 141) dirty exists 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: left oid 7 (ObjNum 141 snap 36 seq_num 141) 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:359: done (1 left) 2024-09-16T02:15:31.938 INFO:tasks.rados.rados.0.smithi195.stdout:363: snap_create 2024-09-16T02:15:31.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:31.973 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:31.973 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:31.973 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:31.973 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:31.974 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:31.979 INFO:tasks.rados.rados.0.smithi195.stdout:362: finishing copy_from racing read to smithi19530025-37 2024-09-16T02:15:31.981 INFO:tasks.rados.rados.0.smithi195.stdout:362: finishing copy_from to smithi19530025-37 2024-09-16T02:15:31.981 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 49 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:31.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:31.998 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:31.994+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:32.098 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:32.097+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:32.121 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:32.121 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:32.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:32.146 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:32.161 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.161 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:32.171 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:32.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:32.199 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:32.198+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:32.200 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:32.200 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:32.228 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:32.228 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:32.257 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:32.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:32.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:32.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:32.297 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.297 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.300 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:32.296+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:32.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:32.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:32.319 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:32.338 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:32.348 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:32.348 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:32.380 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:32.380 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:32.401 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:32.396+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:32.472 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.472 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.480 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.480 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.486 INFO:tasks.thrashosds.thrasher:test_bluestore_reshard completed 2024-09-16T02:15:32.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:32.502 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:32.501+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:32.543 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:32.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:32.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:32.603 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:32.602+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:32.631 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:32.631 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:32.660 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.660 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.673 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:32.673 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:32.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:32.704 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:32.701+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:32.792 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.792 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.805 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:32.802+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:32.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:32.906 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:32.906+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:32.923 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:32.924 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:32.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:32.948 INFO:tasks.rados.rados.0.smithi195.stdout:362: done (1 left) 2024-09-16T02:15:32.948 INFO:tasks.rados.rados.0.smithi195.stdout:363: done (0 left) 2024-09-16T02:15:32.948 INFO:tasks.rados.rados.0.smithi195.stdout:364: read oid 13 snap -1 2024-09-16T02:15:32.948 INFO:tasks.rados.rados.0.smithi195.stdout:364: expect deleted 2024-09-16T02:15:32.949 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 38 current snap is 37 2024-09-16T02:15:32.949 INFO:tasks.rados.rados.0.smithi195.stdout:365: seq_num 143 ranges {0=458752,458752=475136,933888=655360,1589248=671744,2260992=147456} 2024-09-16T02:15:32.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:15:32.956 INFO:tasks.rados.rados.0.smithi195.stdout:365: writing smithi19530025-38 from 0 to 458752 tid 1 2024-09-16T02:15:32.963 INFO:tasks.rados.rados.0.smithi195.stdout:365: writing smithi19530025-38 from 458752 to 933888 tid 2 2024-09-16T02:15:32.973 INFO:tasks.rados.rados.0.smithi195.stdout:365: writing smithi19530025-38 from 933888 to 1589248 tid 3 2024-09-16T02:15:32.983 INFO:tasks.rados.rados.0.smithi195.stdout:365: writing smithi19530025-38 from 1589248 to 2260992 tid 4 2024-09-16T02:15:32.986 INFO:tasks.rados.rados.0.smithi195.stdout:365: writing smithi19530025-38 from 2260992 to 2408448 tid 5 2024-09-16T02:15:32.986 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 31 current snap is 37 2024-09-16T02:15:32.986 INFO:tasks.rados.rados.0.smithi195.stdout:366: seq_num 144 ranges {2031616=16384} 2024-09-16T02:15:33.006 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:33.000+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:366: writing smithi19530025-31 from 2031616 to 2048000 tid 1 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:367: delete oid 46 current snap is 37 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 1 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 2 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 3 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 4 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 5 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 6 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: finishing write tid 7 to smithi19530025-38 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 updating version 0 to 147 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 updating version 147 to 150 2024-09-16T02:15:33.013 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 version 150 is already newer than 148 2024-09-16T02:15:33.014 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 version 150 is already newer than 146 2024-09-16T02:15:33.014 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 updating version 150 to 151 2024-09-16T02:15:33.014 INFO:tasks.rados.rados.0.smithi195.stdout:365: oid 38 version 151 is already newer than 149 2024-09-16T02:15:33.014 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 38 v 151 (ObjNum 143 snap 37 seq_num 143) dirty exists 2024-09-16T02:15:33.014 INFO:tasks.rados.rados.0.smithi195.stdout:365: left oid 38 (ObjNum 143 snap 37 seq_num 143) 2024-09-16T02:15:33.023 INFO:tasks.rados.rados.0.smithi195.stdout:366: finishing write tid 1 to smithi19530025-31 2024-09-16T02:15:33.023 INFO:tasks.rados.rados.0.smithi195.stdout:366: finishing write tid 2 to smithi19530025-31 2024-09-16T02:15:33.026 INFO:tasks.rados.rados.0.smithi195.stdout:366: finishing write tid 3 to smithi19530025-31 2024-09-16T02:15:33.026 INFO:tasks.rados.rados.0.smithi195.stdout:366: oid 31 updating version 0 to 90 2024-09-16T02:15:33.026 INFO:tasks.rados.rados.0.smithi195.stdout:366: oid 31 version 90 is already newer than 89 2024-09-16T02:15:33.026 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 31 v 90 (ObjNum 144 snap 37 seq_num 144) dirty exists 2024-09-16T02:15:33.027 INFO:tasks.rados.rados.0.smithi195.stdout:366: left oid 31 (ObjNum 144 snap 37 seq_num 144) 2024-09-16T02:15:33.054 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:33.054 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:33.071 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.071 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:33.091 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:33.092 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:33.107 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:33.106+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:364: done (3 left) 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:365: done (2 left) 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:366: done (1 left) 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:367: done (0 left) 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:368: read oid 15 snap -1 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:368: expect (ObjNum 124 snap 27 seq_num 124) 2024-09-16T02:15:33.123 INFO:tasks.rados.rados.0.smithi195.stdout:369: snap_remove snap 31 2024-09-16T02:15:33.208 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:33.204+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:33.252 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.252 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:33.308 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:33.306+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:33.409 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:33.405+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:33.431 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.431 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:33.510 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:33.506+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:33.563 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.563 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.578 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:33.610 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:33.604+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:33.709 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.709 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.711 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:33.708+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:33.723 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:33.813 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:33.809+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:33.842 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:33.842 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:33.856 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:33.913 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:33.908+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:33.943 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:15:33.950 INFO:tasks.rados.rados.0.smithi195.stdout:369: done (1 left) 2024-09-16T02:15:33.950 INFO:tasks.rados.rados.0.smithi195.stdout:370: snap_remove snap 26 2024-09-16T02:15:34.015 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:34.010+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:34.025 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.025 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.039 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:34.115 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:34.110+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:34.160 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.160 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:34.217 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:34.217+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:34.292 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.292 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.306 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:34.317 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:34.316+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:34.418 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:34.416+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:34.467 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.467 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.482 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:34.525 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:34.525+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:34.607 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.607 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.619 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:34.617+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:34.621 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:34.720 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:34.717+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:34.782 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.782 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.797 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:34.821 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:34.818+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:34.922 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:34.918+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:34.967 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:34.967 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:34.972 INFO:tasks.rados.rados.0.smithi195.stdout:370: done (1 left) 2024-09-16T02:15:34.972 INFO:tasks.rados.rados.0.smithi195.stdout:371: copy_from oid 42 from oid 48 current snap is 37 2024-09-16T02:15:34.972 INFO:tasks.rados.rados.0.smithi195.stdout:368: expect (ObjNum 124 snap 27 seq_num 124) 2024-09-16T02:15:34.987 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:35.006 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1768152 2024-09-16T02:15:35.023 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:35.016+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:35.048 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 2118795 2024-09-16T02:15:35.053 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:35.081 INFO:tasks.rados.rados.0.smithi195.stdout:371: finishing copy_from racing read to smithi19530025-42 2024-09-16T02:15:35.081 INFO:tasks.rados.rados.0.smithi195.stdout:368: done (1 left) 2024-09-16T02:15:35.081 INFO:tasks.rados.rados.0.smithi195.stdout:372: snap_remove snap 33 2024-09-16T02:15:35.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:35.124 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:35.120+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:35.126 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:35.127 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:35.140 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:35.225 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:35.222+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:35.282 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:35.282 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:35.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:35.307 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:35.307 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:35.308 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:35.322 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:35.326 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:35.326+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:35.337 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:35.338 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:35.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:35.366 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:35.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:35.395 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:35.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:35.427 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:35.425+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 68.103554406000001, 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.334730205, 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.443 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:35.444 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335879+0000", 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335907+0000", 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7291999999999999e-05 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.445 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669030+0000", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33312349400000002 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669045+0000", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4943000000000001e-05 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669245+0000", 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019942899999999999 2024-09-16T02:15:35.446 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669263+0000", 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8445999999999999e-05 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670533+0000", 2024-09-16T02:15:35.447 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001270024 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670602+0000", 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8406999999999997e-05 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:15:35.448 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.104581600000003, 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75881597999999995, 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.449 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334845+0000", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0900000000000001e-07 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334886+0000", 2024-09-16T02:15:35.450 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1254999999999997e-05 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091945+0000", 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75705888700000001 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091960+0000", 2024-09-16T02:15:35.451 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4686e-05 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092157+0000", 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019717000000000001 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092176+0000", 2024-09-16T02:15:35.452 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9020000000000001e-05 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093578+0000", 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001401769 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093660+0000", 2024-09-16T02:15:35.453 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2484000000000005e-05 2024-09-16T02:15:35.454 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.454 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.454 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.454 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.454 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.455 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:35.455 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:35.455 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.087558502, 2024-09-16T02:15:35.455 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017058792, 2024-09-16T02:15:35.455 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.456 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.456 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.456 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.456 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.456 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.457 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:35.457 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.457 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.457 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.457 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.458 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:35.458 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.458 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.458 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.458 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.459 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351868+0000", 2024-09-16T02:15:35.459 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:15:35.459 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.459 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.459 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.460 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351917+0000", 2024-09-16T02:15:35.460 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-09-16T02:15:35.460 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.460 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.460 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.461 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352051+0000", 2024-09-16T02:15:35.461 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013457599999999999 2024-09-16T02:15:35.461 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.461 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.461 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352073+0000", 2024-09-16T02:15:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1489e-05 2024-09-16T02:15:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.462 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352231+0000", 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158455 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.463 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352260+0000", 2024-09-16T02:15:35.464 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8472000000000001e-05 2024-09-16T02:15:35.464 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.464 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.464 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.464 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368854+0000", 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016594647000000001 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368926+0000", 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1534999999999999e-05 2024-09-16T02:15:35.465 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=35, last_complete=45'32, committed=1, applied=1))", 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.057644478, 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65850650099999997, 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.466 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.467 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381783+0000", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.857e-06 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381795+0000", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1466e-05 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040081+0000", 2024-09-16T02:15:35.468 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65828677099999999 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040095+0000", 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3686000000000001e-05 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040215+0000", 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011949 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.469 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040288+0000", 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3231000000000004e-05 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:35.470 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.042438384999997, 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27792197099999999, 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.471 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396988+0000", 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759999999999999e-06 2024-09-16T02:15:35.472 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397010+0000", 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1525000000000001e-05 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673630+0000", 2024-09-16T02:15:35.473 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27662027 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673655+0000", 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5003000000000001e-05 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673857+0000", 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020175299999999999 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.474 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.475 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.475 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673875+0000", 2024-09-16T02:15:35.475 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:15:35.475 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.475 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.476 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.476 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674832+0000", 2024-09-16T02:15:35.476 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095631999999999998 2024-09-16T02:15:35.476 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.476 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.477 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.477 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674909+0000", 2024-09-16T02:15:35.477 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7741e-05 2024-09-16T02:15:35.477 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.477 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.478 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.478 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.478 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.478 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:487 3.7s0 3:ed091bc8:::smithi19530025-23:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc d=[d,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:35.478 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.506812384, 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019918874, 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:35.479 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:35.480 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:35.480 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:35.480 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 487 2024-09-16T02:15:35.481 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.481 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.481 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.481 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.481 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:35.482 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.482 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.482 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.482 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.482 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.483 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932617+0000", 2024-09-16T02:15:35.484 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.309e-06 2024-09-16T02:15:35.484 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.484 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.484 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932695+0000", 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8548999999999996e-05 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932736+0000", 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0349000000000003e-05 2024-09-16T02:15:35.485 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932757+0000", 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1903e-05 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932928+0000", 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017018600000000001 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.486 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932950+0000", 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2395000000000002e-05 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934568+0000", 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00161812 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:35.487 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934773+0000", 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000204995 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936578+0000", 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001804709 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.940066+0000", 2024-09-16T02:15:35.488 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034883869999999999 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952401+0000", 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012334539 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952532+0000", 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131433 2024-09-16T02:15:35.489 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.498410608, 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013412999, 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.490 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.491 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941016+0000", 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6000000000000004e-07 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941065+0000", 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9462000000000002e-05 2024-09-16T02:15:35.492 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941113+0000", 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941128+0000", 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4348999999999999e-05 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.493 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941254+0000", 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012600400000000001 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941277+0000", 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3482999999999999e-05 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.494 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954333+0000", 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013055430999999999 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954428+0000", 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5468000000000001e-05 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.495 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=50, reqid=client.4598.0:494, at_version=58'42, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 48.480479662, 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010360967, 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.496 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958946+0000", 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.497 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.958947+0000", 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4300000000000003e-07 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959048+0000", 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000101671 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959143+0000", 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5120999999999999e-05 2024-09-16T02:15:35.498 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959174+0000", 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0803000000000003e-05 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959376+0000", 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000202105 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.499 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.959404+0000", 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7141999999999999e-05 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969241+0000", 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098377399999999993 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.500 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.969307+0000", 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5741999999999995e-05 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=51, reqid=client.4598.0:495, at_version=58'43, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 48.473747443000001, 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011275655000000001, 2024-09-16T02:15:35.501 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965678+0000", 2024-09-16T02:15:35.502 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965681+0000", 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2199999999999999e-06 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965731+0000", 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0429000000000001e-05 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.503 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965878+0000", 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014756 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965904+0000", 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5686000000000001e-05 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.504 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967710+0000", 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001805652 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.968220+0000", 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050980399999999999 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976848+0000", 2024-09-16T02:15:35.505 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0086280009999999997 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976954+0000", 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000106303 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.506 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 58/32 ECSubWrite(tid=52, reqid=client.4598.0:496, at_version=58'44, trim_to=0'0, roll_forward_to=57'40))", 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 48.473683401000002, 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011274895, 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:35.507 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965742+0000", 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965743+0000", 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1069999999999999e-06 2024-09-16T02:15:35.508 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965783+0000", 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9165999999999997e-05 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967647+0000", 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0018640340000000001 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.509 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967671+0000", 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4712e-05 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971326+0000", 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036543130000000002 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.510 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.971353+0000", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7376e-05 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976963+0000", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056098989999999998 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.977017+0000", 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4287999999999999e-05 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.511 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=86, reqid=osd.6.0:86, at_version=73'70, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.49417802, 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015494396000000001, 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.512 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945254+0000", 2024-09-16T02:15:35.513 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3130000000000004e-06 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945315+0000", 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0584999999999997e-05 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945408+0000", 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3288999999999994e-05 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.514 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945425+0000", 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7329e-05 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945587+0000", 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016137399999999999 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945615+0000", 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8048999999999999e-05 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.515 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959667+0000", 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014052007 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.960742+0000", 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010754499999999999 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.516 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=87, reqid=osd.6.0:87, at_version=73'72, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.491348154000001, 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020784545000000001, 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.517 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948079+0000", 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.877e-06 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.518 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948132+0000", 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2312000000000003e-05 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948237+0000", 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105015 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948259+0000", 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2668999999999999e-05 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.519 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948410+0000", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015014100000000001 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948436+0000", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6060000000000001e-05 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968776+0000", 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020339946000000001 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.520 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968862+0000", 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6525000000000003e-05 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs5 73/32 ECSubWrite(tid=80, reqid=osd.14.0:80, at_version=73'29, trim_to=0'0, roll_forward_to=70'27))", 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.476705881999997, 2024-09-16T02:15:35.521 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019480810000000001, 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.522 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962721+0000", 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-07 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962770+0000", 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9233000000000001e-05 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.523 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962872+0000", 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010231800000000001 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962901+0000", 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8765999999999999e-05 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963118+0000", 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021719600000000001 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.524 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963156+0000", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7435999999999999e-05 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982116+0000", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018960226 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982201+0000", 2024-09-16T02:15:35.525 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4818e-05 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=133, reqid=osd.11.0:133, at_version=73'129, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.473630460999999, 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016499698, 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.526 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.17e-07 2024-09-16T02:15:35.527 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965821+0000", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.569e-05 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965896+0000", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5185999999999995e-05 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965907+0000", 2024-09-16T02:15:35.528 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0379e-05 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966895+0000", 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098857000000000003 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966922+0000", 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7066000000000001e-05 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.529 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982221+0000", 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015298589 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982295+0000", 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4000999999999998e-05 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=134, reqid=osd.11.0:134, at_version=73'131, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:35.530 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.473602526999997, 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017078139999999999, 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.531 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.03e-07 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965841+0000", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7326e-05 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.532 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965939+0000", 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8420999999999998e-05 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965956+0000", 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6988999999999999e-05 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972222+0000", 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062658890000000002 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.533 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972246+0000", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4403e-05 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982860+0000", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010613734 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982901+0000", 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1174999999999998e-05 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.534 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=88, reqid=osd.6.0:87, at_version=0'0, trim_to=0'0, roll_forward_to=73'72))", 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 33.468758370000003, 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012896283, 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.535 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970667+0000", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970668+0000", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.7199999999999997e-07 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.536 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970682+0000", 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.322e-05 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970722+0000", 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0429000000000002e-05 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970732+0000", 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0105999999999999e-05 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.537 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.971118+0000", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038623200000000002 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.971142+0000", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.983483+0000", 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012341193 2024-09-16T02:15:35.538 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.983564+0000", 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0289000000000003e-05 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 74/69 ECSubWrite(tid=137, reqid=osd.11.0:137, at_version=74'135, trim_to=0'0, roll_forward_to=73'131))", 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 32.460450131999998, 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010379599999999999, 2024-09-16T02:15:35.539 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978976+0000", 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.540 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.978977+0000", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8670000000000001e-06 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979030+0000", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2447000000000001e-05 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979134+0000", 2024-09-16T02:15:35.541 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000103835 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979156+0000", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2558999999999999e-05 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979320+0000", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000163738 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.979347+0000", 2024-09-16T02:15:35.542 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7282000000000001e-05 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.989264+0000", 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0099165069999999998 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:02.989355+0000", 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1365000000000004e-05 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.543 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:724 3.7s0 3:ebce791d:::smithi19530025-35:head [append 0~0] snapc 18=[18,d] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e81)", 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 25.421764500999998, 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013022176999999999, 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 724 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.544 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017661+0000", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017662+0000", 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9400000000000004e-07 2024-09-16T02:15:35.545 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017666+0000", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8999999999999999e-06 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017676+0000", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0386e-05 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.017690+0000", 2024-09-16T02:15:35.546 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3502999999999999e-05 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.024290+0000", 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.006600318 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.024505+0000", 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021444500000000001 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.547 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.025313+0000", 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00080815300000000004 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.027220+0000", 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001907066 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.030639+0000", 2024-09-16T02:15:35.548 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034191949999999999 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:10.030683+0000", 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4416999999999998e-05 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:788 3.7s0 3:efbf0545:::smithi19530025-32:head [create,setxattr V\\UcZkfoVZbifk^fW\\TngQQUPi`QcehgZRpbncho (1999) in=40b,setxattr ZRreckT_dVf\\ctRT_i^dTRadaj\\ooXkpWhUg^c (4068) in=38b,setxattr ]O^Y`oiq\\set`eYfOpZ]OqpafSeUrceob[jghefP (873) in=40b,setxattr pa`OlQbdkPQV`habZkeb\\UaZhqri`qiXYgqmt_cc (1542) in=40b,setxattr OVbZnokgr`S_rPkXal\\moopUTdp[rrRYi]Oi]bpg (1297) in=40b,setxattr TRpko\\X\\jsnarS\\dkqaY\\c\\`aiVlhot[[o_O_UoR (1684) in=40b,setxattr Urbt (4068) in=4b,setxattr ^][Unhjgnmr[nWlcpqqhss__fd[WOUToconim]Pj (3084) in=40b,setxattr iU[ard`mj[Qijanah`QONRPPPPWnjV]dZmoVlOOP (1262) in=40b,setxattr O\\YgbW\\hbbUsUSYfg\\TToafkkSjn]ZRYaVQkiQme (245) in=40b,setxattr XqWdb^p[gN[RgrQQYdgqdbUS`aqdSTh_herpkti[ (173) in=40b,setxattr pWY`hiOY`QdPYT\\iOQpUiTUUXckkmUc_rkXiVOjs (699) in=40b,setxattr ncTPYl^j\\dpmpb]TiNUe`j]mohmqia\\NstbaRRgV (1927) in=40b,setxattr r^WOa`]_fNedaNN_edhqXf`eRUfUQi\\aoTWeX\\a_ (16140) in=40b,setxattr ]bbomOdb_dN_\\SYQdOr]OieQoahTqgrr\\llq\\[dg (12488) in=40b,setxattr aPcNrlTpY`tflOtRiY_UYgScbli]d^ZWQeSRsSTr (525) in=40b,setxattr XWk\\TgTZfTp`Znlj_N\\a[^cYtdocP`cYfO\\dUY`^ (490) in=40b,setxattr SabROtbgPZ[^aaQ^lVRWgQaToRknOSgk`gepYcVP (17332) in=40b,setxattr YSPXXlaUgoelQOSTk`rYfP_jsmZZqblXaeegO_cg (1329) in=40b,setxattr dphWftYOkiffV[XU^WqtifdadXcWlj (4068) in=30b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.444455000000001, 2024-09-16T02:15:35.549 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012031088000000001, 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 788 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.550 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994971+0000", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.994973+0000", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5780000000000001e-06 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995089+0000", 2024-09-16T02:15:35.551 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000115374 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995112+0000", 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3201e-05 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995124+0000", 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2421e-05 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.552 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995232+0000", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000107409 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995396+0000", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000164573 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.996898+0000", 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001501549 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.553 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.003436+0000", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065382130000000002 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.006853+0000", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003416592 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.007002+0000", 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000149178 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.554 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.6.22:4 3.7s0 3:ed091bc8:::smithi19530025-23:head [assert-version v49,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 3.4875911880000001, 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017925606, 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:35.555 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951838+0000", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7000000000000002e-06 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951854+0000", 2024-09-16T02:15:35.556 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5563e-05 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951902+0000", 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8402000000000002e-05 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951924+0000", 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1469000000000001e-05 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:35.557 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.952770+0000", 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00084661899999999995 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.953027+0000", 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025700000000000001 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.969760+0000", 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016732852999999999 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:35.558 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:35.559 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:35.559 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:35.560 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:35.561 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:35.561 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:35.561 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:35.562 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:35.557+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:35.661 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:35.656+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:35.687 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:35.688 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:35.688 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:35.688 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:35.701 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:35.725 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:35.726 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:35.762 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:35.758+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:35.770 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:35.770 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:35.815 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:35.815 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:35.825 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:35.826 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:35.826 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:35.826 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:35.826 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:35.840 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:35.857 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:35.863 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:35.861+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:35.894 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:35.894 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:372: done (1 left) 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:373: rollback oid 9 current snap is 37 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 9 to 30 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:374: rollback oid 23 current snap is 37 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 23 to 27 2024-09-16T02:15:35.923 INFO:tasks.rados.rados.0.smithi195.stdout:375: read oid 15 snap -1 2024-09-16T02:15:35.924 INFO:tasks.rados.rados.0.smithi195.stdout:375: expect (ObjNum 124 snap 27 seq_num 124) 2024-09-16T02:15:35.924 INFO:tasks.rados.rados.0.smithi195.stdout:376: setattr oid 12 current snap is 37 2024-09-16T02:15:35.924 INFO:tasks.rados.rados.0.smithi195.stdout:377: copy_from oid 37 from oid 18 current snap is 37 2024-09-16T02:15:35.924 INFO:tasks.rados.rados.0.smithi195.stdout:378: read oid 45 snap -1 2024-09-16T02:15:35.925 INFO:tasks.rados.rados.0.smithi195.stdout:378: expect (ObjNum 75 snap 8 seq_num 75) 2024-09-16T02:15:35.925 INFO:tasks.rados.rados.0.smithi195.stdout:379: rollback oid 8 current snap is 37 2024-09-16T02:15:35.925 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 8 to 27 2024-09-16T02:15:35.925 INFO:tasks.rados.rados.0.smithi195.stdout:380: snap_remove snap 36 2024-09-16T02:15:35.949 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:35.964 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:35.962+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:35.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:15:35.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:35.982+0000 7f28be7fc640 0 -- 172.21.15.3:0/2682185005 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f28c02242f0 msgr2=0x7f28c02266e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:35.993 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:484 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [setxattr _header (57) in=7b] snapc d=[d,c,a] ondisk+write+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:15:36.025 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.015892848999997, 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62580566599999998, 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 484 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.026 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006735+0000", 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006736+0000", 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2500000000000004e-07 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.027 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006740+0000", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6460000000000001e-06 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006749+0000", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0390000000000005e-06 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006754+0000", 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1939999999999997e-06 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.028 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011015+0000", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042612830000000003 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011096+0000", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0894000000000002e-05 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.011534+0000", 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00043832300000000001 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.029 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012404+0000", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086940800000000001 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632372+0000", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.619968348 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632541+0000", 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016870599999999999 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.030 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:485 3.bs0 3:dfc9a9cf:::smithi19530025-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e56)", 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.015866475999999, 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.628869487, 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 485 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.031 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006761+0000", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006762+0000", 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4499999999999997e-07 2024-09-16T02:15:36.032 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006765+0000", 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9270000000000001e-06 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006771+0000", 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3679999999999998e-06 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.033 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.006777+0000", 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2959999999999998e-06 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012433+0000", 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056561729999999996 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012499+0000", 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6303999999999994e-05 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.034 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633369+0000", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62087044899999999 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633608+0000", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023843099999999999 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.635631+0000", 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020229940000000002 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.035 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 56/32 ECSubWriteReply(tid=59, last_complete=56'17, committed=1, applied=1))", 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 51.009899955000002, 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.62052908399999995, 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.036 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012728+0000", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012730+0000", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.694e-06 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.012741+0000", 2024-09-16T02:15:36.037 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1343e-05 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632045+0000", 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.61930391299999998 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632066+0000", 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0699999999999998e-05 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.038 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.632237+0000", 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017166600000000001 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.633257+0000", 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010197680000000001 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:497 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.066409036000003, 2024-09-16T02:15:36.039 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028762988, 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 497 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:36.040 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956219+0000", 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956222+0000", 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6200000000000001e-06 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.041 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956239+0000", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6458999999999999e-05 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956280+0000", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0942999999999997e-05 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956303+0000", 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2742999999999999e-05 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.042 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956495+0000", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000192762 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956517+0000", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1375000000000001e-05 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962337+0000", 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0058203600000000001 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.043 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.962579+0000", 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00024195400000000001 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.984982+0000", 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022402772000000001 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.044 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:498 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.066277513000003, 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030739767000000001, 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 498 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:36.045 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956350+0000", 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956352+0000", 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.75e-06 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.046 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956362+0000", 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6900000000000004e-06 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956408+0000", 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5828999999999998e-05 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956426+0000", 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8216e-05 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.047 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956684+0000", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025823399999999998 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956702+0000", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963967+0000", 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0072649940000000003 2024-09-16T02:15:36.048 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.966462+0000", 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0024953639999999999 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.987090+0000", 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020627774000000001 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.049 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:499 3.bs0 3:d814c48f:::smithi19530025-7:head [read 0~1622016,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.066172551999998, 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034753205000000002, 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 499 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.050 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956455+0000", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956457+0000", 2024-09-16T02:15:36.051 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5969999999999999e-06 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956469+0000", 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1929999999999999e-05 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956493+0000", 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3849000000000002e-05 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.052 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956509+0000", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6333000000000001e-05 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956872+0000", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000363424 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956891+0000", 2024-09-16T02:15:36.053 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8913e-05 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967274+0000", 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010382466 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.967524+0000", 2024-09-16T02:15:36.054 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025058799999999999 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.991209+0000", 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023684105 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:500 3.4s0 3:23d2836d:::smithi19530025-21:head [rmxattr XftrktNgjnZekN[]^ls\\[htgSaXkUrlVOqS[WcNt in=40b,rmxattr YgOToR[hPbQocWaoh\\h_NkYaghbpdean`\\kalgQ[ in=40b,rmxattr b_e\\_i^rj`csr^q^\\e_boR\\lQNhcZg_NVqpgUQ`Q in=40b,rmxattr qjofnWtf]NipeS]\\gcjdaepnS_Pi^QgbRZd in=35b] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:36.055 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.066048891000001, 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030187572999999999, 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 500 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.056 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956579+0000", 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956581+0000", 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6959999999999999e-06 2024-09-16T02:15:36.057 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956591+0000", 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.025e-05 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956614+0000", 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3068999999999999e-05 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.058 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956632+0000", 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8199999999999999e-05 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956786+0000", 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015396999999999999 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956813+0000", 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6872e-05 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.059 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963384+0000", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0065708399999999997 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.963658+0000", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00027420299999999999 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.965812+0000", 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021535809999999999 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.060 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.981888+0000", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016076136000000001 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986582+0000", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046937309999999996 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.986767+0000", 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018502500000000001 2024-09-16T02:15:36.061 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 61/60 ECSubWriteReply(tid=78, last_complete=61'24, committed=1, applied=1))", 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 45.833679777, 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45998103899999998, 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.062 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188948+0000", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188949+0000", 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5700000000000002e-07 2024-09-16T02:15:36.063 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.188953+0000", 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.5129999999999998e-06 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647595+0000", 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.458642198 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.064 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647616+0000", 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0886e-05 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.647777+0000", 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016024999999999999 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:50.648929+0000", 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011525350000000001 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.065 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.bs0 71/32 ECSubWriteReply(tid=99, last_complete=71'25, committed=1, applied=1))", 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.137834658999999, 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43656151100000001, 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.066 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884793+0000", 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884795+0000", 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7230000000000001e-06 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.067 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:00.884807+0000", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1683e-05 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320340+0000", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43553291399999999 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320354+0000", 2024-09-16T02:15:36.068 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4953000000000001e-05 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.320476+0000", 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000121335 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.321355+0000", 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000878903 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.069 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:646 3.4s0 3:2f4f4374:::smithi19530025-31:head [append 0~655360] snapc 14=[13,12,d] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.066821625999999, 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031432506999999998, 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.070 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 646 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955806+0000", 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.071 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955808+0000", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9379999999999999e-06 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956651+0000", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00084241399999999999 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956666+0000", 2024-09-16T02:15:36.072 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5089000000000001e-05 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956674+0000", 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3680000000000002e-06 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956762+0000", 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7546000000000005e-05 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.073 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956890+0000", 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012829200000000001 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958306+0000", 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001415735 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961177+0000", 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0028716689999999999 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.074 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987150+0000", 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025972928999999999 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987239+0000", 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8527000000000001e-05 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:647 3.4s0 3:2f4f4374:::smithi19530025-31:head [append 0~606208] snapc 14=[13,12,d] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-16T02:15:36.075 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 34.061419659000002, 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026701157, 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 647 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.076 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961208+0000", 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961211+0000", 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5239999999999999e-06 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.077 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961712+0000", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00050093999999999996 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961728+0000", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6379999999999999e-05 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961739+0000", 2024-09-16T02:15:36.078 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1090999999999999e-05 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.961947+0000", 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020735 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962184+0000", 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023711199999999999 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.079 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964605+0000", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002421154 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.970291+0000", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056862900000000001 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987855+0000", 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017563552999999999 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.080 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.987909+0000", 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4762999999999997e-05 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:708 3.bs0 3:d29cad4f:::smithi19530025-29:head [setxattr _header (57) in=7b] snapc 16=[d] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 27.989984078999999, 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029424117999999999, 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.081 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 708 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.082 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032644+0000", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4300000000000003e-07 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032650+0000", 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1229999999999999e-06 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.083 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032654+0000", 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7310000000000001e-06 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032658+0000", 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.6049999999999998e-06 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.084 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.034706+0000", 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020481869999999999 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.034795+0000", 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9376000000000006e-05 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035215+0000", 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041973200000000003 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.085 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035984+0000", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000768598 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061988+0000", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026003727000000001 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062068+0000", 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0395999999999994e-05 2024-09-16T02:15:36.086 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:709 3.bs0 3:d29cad4f:::smithi19530025-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 27.989964634, 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.030374595000000001, 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.087 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 709 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032663+0000", 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.088 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032664+0000", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.77e-07 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032666+0000", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0049999999999999e-06 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032672+0000", 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.516e-06 2024-09-16T02:15:36.089 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032678+0000", 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.3730000000000001e-06 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035885+0000", 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0032076069999999999 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.090 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.035962+0000", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6895000000000002e-05 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062474+0000", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.026512285 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062564+0000", 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9628000000000003e-05 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.091 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.063038+0000", 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00047390900000000001 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:719 3.4s0 3:2796292c:::smithi19530025-3:head [copy-from ver 21] snapc 16=[d] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 27.989942156000001, 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.032989046000000001, 2024-09-16T02:15:36.092 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 719 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.093 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032686+0000", 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4799999999999999e-07 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.094 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032689+0000", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.368e-06 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032693+0000", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4750000000000004e-06 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032700+0000", 2024-09-16T02:15:36.095 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.6030000000000001e-06 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032819+0000", 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000119722 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033058+0000", 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000239101 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.096 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.057868+0000", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024809127 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.062005+0000", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0041371569999999998 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065628+0000", 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036227859999999998 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.097 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065675+0000", 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.7259000000000003e-05 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:720 3.4s0 3:2796292c:::smithi19530025-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 27.989913637000001, 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.029151192999999999, 2024-09-16T02:15:36.098 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 720 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.099 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032714+0000", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032715+0000", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.75e-07 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032718+0000", 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5930000000000002e-06 2024-09-16T02:15:36.100 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032727+0000", 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2369999999999992e-06 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.032736+0000", 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3610000000000007e-06 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.101 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033456+0000", 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072037800000000003 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.033521+0000", 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5143999999999998e-05 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061586+0000", 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.028064787000000001 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.102 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061711+0000", 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012470999999999999 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.061865+0000", 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015450800000000001 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.103 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.4s0 79/66 ECSubWriteReply(tid=150, last_complete=79'55, committed=1, applied=1))", 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 27.955087268, 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.39361127800000001, 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.104 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067541+0000", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.7199999999999998e-07 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.067548+0000", 2024-09-16T02:15:36.105 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.2319999999999998e-06 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.460948+0000", 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.39340061199999998 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.460961+0000", 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.311e-05 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.106 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.461117+0000", 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015610900000000001 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.461152+0000", 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4542999999999998e-05 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:764 3.4s0 3:2f4f4374:::smithi19530025-31:head [setxattr _header (57) in=7b] snapc 19=[18,d] ondisk+write+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:36.107 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 21.933647786999998, 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.60102845199999999, 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 764 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.108 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088980+0000", 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088981+0000", 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5499999999999998e-07 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.109 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088984+0000", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5990000000000002e-06 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088991+0000", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.148e-06 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.088997+0000", 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0459999999999996e-06 2024-09-16T02:15:36.110 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092231+0000", 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003234823 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092325+0000", 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3385000000000001e-05 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.111 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.092707+0000", 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00038185900000000002 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093576+0000", 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00086912100000000004 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.689836+0000", 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.59626043900000003 2024-09-16T02:15:36.112 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690009+0000", 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017237700000000001 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:765 3.4s0 3:2f4f4374:::smithi19530025-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 21.933622998000001, 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.60354616400000005, 2024-09-16T02:15:36.113 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 765 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.114 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089005+0000", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8999999999999997e-07 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089008+0000", 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5809999999999999e-06 2024-09-16T02:15:36.115 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089016+0000", 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1689999999999992e-06 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.089023+0000", 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4339999999999997e-06 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.116 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093437+0000", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044148850000000003 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.093522+0000", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4493999999999994e-05 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690813+0000", 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.59729147199999999 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.117 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.690957+0000", 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014378899999999999 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:14.692551+0000", 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00159385 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.118 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s1 90/32 ECSubWrite(tid=127, reqid=client.4598.0:796, at_version=90'95, trim_to=0'0, roll_forward_to=90'91))", 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 16.992246001000002, 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.43142370499999999, 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.119 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030382+0000", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030383+0000", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.235e-06 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.030470+0000", 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7199000000000003e-05 2024-09-16T02:15:36.120 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460364+0000", 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.42989403399999998 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460379+0000", 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.448e-05 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.121 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460489+0000", 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011054599999999999 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.460512+0000", 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2736000000000001e-05 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.461736+0000", 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001223937 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.122 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.461806+0000", 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9537999999999997e-05 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.9s0 95/54 ECSubWriteReply(tid=197, last_complete=95'22, committed=1, applied=1))", 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:24.043314+0000", 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.979314110000001, 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79681145200000003, 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.123 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.043314+0000", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.043314+0000", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.043315+0000", 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.54e-07 2024-09-16T02:15:36.124 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.043319+0000", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9540000000000003e-06 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.838812+0000", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79549300599999995 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.838828+0000", 2024-09-16T02:15:36.125 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5619000000000001e-05 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.838992+0000", 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000163919 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:24.840125+0000", 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0011332 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.126 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.127 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:36.128 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:36.126+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:36.128 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:36.226 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:36.226 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:36.228 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:36.226+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:36.300 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:36.301 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:36.301 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:36.301 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:36.316 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:36.364 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:36.362+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:36.430 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:36.426+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:36.437 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:36.437 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:36.437 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:36.438 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:36.438 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:36.452 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:36.540 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:36.538+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:36.623 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:36.623 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:36.623 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:36.623 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:36.623 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:33 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.328538409000004, 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024514080000000001, 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 33 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.624 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291063+0000", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291064+0000", 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.1999999999999999e-07 2024-09-16T02:15:36.625 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291067+0000", 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7199999999999998e-06 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291074+0000", 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4680000000000003e-06 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.626 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291080+0000", 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.964e-06 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0055391950000000002 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0838000000000001e-05 2024-09-16T02:15:36.627 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313405+0000", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016704404999999999 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.313545+0000", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014004999999999999 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315578+0000", 2024-09-16T02:15:36.628 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020328199999999999 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:39 3.1s0 3:81c6c0a8:::smithi19530025-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.328168543999993, 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024374809000000001, 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.629 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 39 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291433+0000", 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.630 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291434+0000", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291440+0000", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.4960000000000004e-06 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291456+0000", 2024-09-16T02:15:36.631 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025e-05 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291468+0000", 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.257e-05 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299421+0000", 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0079526309999999999 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.632 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299510+0000", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.9506000000000007e-05 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314495+0000", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014984548 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.314572+0000", 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6854999999999995e-05 2024-09-16T02:15:36.633 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315808+0000", 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012362639999999999 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:40 3.1s0 3:830a71a8:::smithi19530025-14:head [append 0~229376,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.328116952000002, 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024497693000000001, 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.634 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 40 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291485+0000", 2024-09-16T02:15:36.635 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291486+0000", 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8999999999999995e-07 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291773+0000", 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028741999999999998 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.636 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291795+0000", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.181e-05 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291807+0000", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1687e-05 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299547+0000", 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0077406699999999998 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.637 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.299647+0000", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9978999999999999e-05 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.300445+0000", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00079766400000000001 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302061+0000", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016158539999999999 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.638 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315913+0000", 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013852573999999999 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.315983+0000", 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9145000000000003e-05 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:41 3.1s0 3:830a71a8:::smithi19530025-14:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:36.639 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.327769618000005, 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024477925000000001, 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 41 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.640 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291832+0000", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291833+0000", 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0300000000000001e-06 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.641 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291839+0000", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.7620000000000001e-06 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291855+0000", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6011e-05 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291865+0000", 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0037999999999999e-05 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.642 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301194+0000", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0093284930000000002 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301281+0000", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6972999999999996e-05 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.301794+0000", 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00051343099999999996 2024-09-16T02:15:36.643 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.304892+0000", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030976950000000001 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316223+0000", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011331071999999999 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.316310+0000", 2024-09-16T02:15:36.644 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.742e-05 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:42 3.1s0 3:830a71a8:::smithi19530025-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.327723426000006, 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031481841000000003, 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.645 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 42 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291878+0000", 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.646 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291880+0000", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2139999999999999e-06 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291883+0000", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.8160000000000004e-06 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291902+0000", 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8881000000000001e-05 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.647 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.291912+0000", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9680000000000001e-06 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302284+0000", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010371521999999999 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.302334+0000", 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.0047999999999999e-05 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.648 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.318985+0000", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016650888999999999 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.319121+0000", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000136336 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.323360+0000", 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0042391670000000003 2024-09-16T02:15:36.649 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:247 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (56) in=7b] snapc 6=[2] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 63.189040614, 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30233062700000002, 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.650 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 247 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430561+0000", 2024-09-16T02:15:36.651 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430562+0000", 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1230000000000001e-06 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430568+0000", 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.519e-06 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.652 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430582+0000", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.366e-05 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430592+0000", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0482999999999999e-05 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433637+0000", 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003044705 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.653 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.433734+0000", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6916000000000005e-05 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434127+0000", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00039313800000000002 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435690+0000", 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015630640000000001 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.654 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732747+0000", 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29705759700000001 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.732892+0000", 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014442200000000001 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:248 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-16T02:15:36.655 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 63.188966170999997, 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.304991454, 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.656 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 248 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430636+0000", 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.657 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430637+0000", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3099999999999996e-07 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430641+0000", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.9090000000000002e-06 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430654+0000", 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3638999999999999e-05 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.658 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.430662+0000", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7370000000000006e-06 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.434972+0000", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0043098019999999997 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.435055+0000", 2024-09-16T02:15:36.659 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3625e-05 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.733794+0000", 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.298738691 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.734027+0000", 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023328199999999999 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.660 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.735627+0000", 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015998379999999999 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:250 3.1s0 3:89ce0a26:::smithi19530025-11:head [copy-from ver 2] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.274541419000002, 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.046806635999999999, 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.661 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 250 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345060+0000", 2024-09-16T02:15:36.662 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345064+0000", 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.484e-06 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345079+0000", 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4939999999999999e-05 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.663 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345114+0000", 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.5258e-05 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345131+0000", 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7356000000000001e-05 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345312+0000", 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018022200000000001 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.664 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345332+0000", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0509e-05 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.356859+0000", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011526471 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.357011+0000", 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015221500000000001 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.665 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.375994+0000", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018982974999999999 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.380796+0000", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0048023010000000001 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391742+0000", 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010946141 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.666 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.391867+0000", 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000124764 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:251 3.1s0 3:89ce0a26:::smithi19530025-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e45)", 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.274423812999999, 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.034174018, 2024-09-16T02:15:36.667 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 251 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.668 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345178+0000", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345179+0000", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1799999999999999e-06 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345186+0000", 2024-09-16T02:15:36.669 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5259999999999999e-06 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345202+0000", 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6453999999999999e-05 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345213+0000", 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1147e-05 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.670 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345556+0000", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000342219 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.345581+0000", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5876000000000001e-05 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358047+0000", 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012465654 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.671 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.358190+0000", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014254599999999999 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379056+0000", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020865887 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379212+0000", 2024-09-16T02:15:36.672 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156675 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.379352+0000", 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013985399999999999 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.1s0 45/32 ECSubWriteReply(tid=55, last_complete=45'49, committed=1, applied=1))", 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 62.226604797, 2024-09-16T02:15:36.673 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.456257309, 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392997+0000", 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.674 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.392998+0000", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2780000000000001e-06 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.393006+0000", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1310000000000006e-06 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847561+0000", 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.45455489999999998 2024-09-16T02:15:36.675 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847576+0000", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.453e-05 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.847695+0000", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000118799 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.849254+0000", 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015596710000000001 2024-09-16T02:15:36.676 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:260 3.1s0 3:830a71a8:::smithi19530025-14:head [copy-from ver 2] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.262493231000001, 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.02650228, 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.677 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 260 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357109+0000", 2024-09-16T02:15:36.678 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357110+0000", 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.243e-06 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357117+0000", 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.9e-06 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.679 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357132+0000", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5156e-05 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357141+0000", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3999999999999998e-06 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357434+0000", 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029301799999999998 2024-09-16T02:15:36.680 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357457+0000", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2821999999999999e-05 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365347+0000", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.007890088 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365650+0000", 2024-09-16T02:15:36.681 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00030261099999999999 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.379802+0000", 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014151762 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381743+0000", 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001941531 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.682 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383415+0000", 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016721349999999999 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.383611+0000", 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019561399999999999 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:261 3.1s0 3:830a71a8:::smithi19530025-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.262412404000003, 2024-09-16T02:15:36.683 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024496032000000001, 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 261 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.684 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357189+0000", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357191+0000", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3799999999999999e-06 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357198+0000", 2024-09-16T02:15:36.685 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1740000000000003e-06 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357220+0000", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2322000000000001e-05 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357238+0000", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7377e-05 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357578+0000", 2024-09-16T02:15:36.686 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00034031900000000001 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357600+0000", 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1543e-05 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366106+0000", 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0085068360000000003 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:36.687 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.366241+0000", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013419899999999999 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381121+0000", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014880769 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381357+0000", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023564300000000001 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.688 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.381685+0000", 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00032846999999999998 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:502 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1658552~160072,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.662607239000003, 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.025237151999999999, 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.689 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 502 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956995+0000", 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.690 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.956996+0000", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.28e-06 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957005+0000", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8459999999999997e-06 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957029+0000", 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4053000000000002e-05 2024-09-16T02:15:36.691 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957045+0000", 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6293000000000001e-05 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957336+0000", 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00029122600000000001 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.692 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957354+0000", 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7543000000000001e-05 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974088+0000", 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016734532999999999 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.974356+0000", 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026803599999999998 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.693 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.982232+0000", 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0078753420000000005 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:503 3.1s0 3:89ce0a26:::smithi19530025-11:head [read 1451430~367194,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.662482029000003, 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.32090356199999998, 2024-09-16T02:15:36.694 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 503 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.695 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957120+0000", 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957122+0000", 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3470000000000001e-06 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.696 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957136+0000", 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4079e-05 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957168+0000", 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2159000000000002e-05 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957189+0000", 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0367e-05 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.697 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957429+0000", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000240561 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.957446+0000", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6472000000000001e-05 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976521+0000", 2024-09-16T02:15:36.698 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019075392 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.976775+0000", 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025338100000000001 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278023+0000", 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.30124880399999998 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.699 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpReadReply(3.1s0 58/32 ECSubReadReply(tid=123, attrs_read=0))", 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 49.641127679999997, 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29957913899999999, 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:36.700 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978474+0000", 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978475+0000", 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.2999999999999999e-07 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.701 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:46.978563+0000", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8211e-05 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277175+0000", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.29861169199999998 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277197+0000", 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1566999999999999e-05 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.702 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.277332+0000", 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013569999999999999 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:47.278053+0000", 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00072103900000000003 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.703 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es4 86/83 ECSubWrite(tid=95, reqid=client.4598.0:767, at_version=0'0, trim_to=0'0, roll_forward_to=86'56))", 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 21.519360125999999, 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83795019199999998, 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.704 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100242+0000", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100257+0000", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5393e-05 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.100278+0000", 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0883e-05 2024-09-16T02:15:36.705 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936352+0000", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.83607375399999995 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936368+0000", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6503000000000001e-05 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936508+0000", 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013982799999999999 2024-09-16T02:15:36.706 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.936525+0000", 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7042999999999999e-05 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.938115+0000", 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001589795 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.707 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.938192+0000", 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6993e-05 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:777 3.1s0 3:8203d376:::smithi19530025-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e87)", 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 20.640514811999999, 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.023317917000000001, 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.708 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 777 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.709 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979087+0000", 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979089+0000", 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1299999999999999e-06 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979098+0000", 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.3529999999999995e-06 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.710 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979121+0000", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3329e-05 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979143+0000", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1656000000000001e-05 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979310+0000", 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000167744 2024-09-16T02:15:36.711 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.979325+0000", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4363e-05 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985273+0000", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0059480979999999998 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985511+0000", 2024-09-16T02:15:36.712 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000238748 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.001858+0000", 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016346612999999999 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.002120+0000", 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026181300000000001 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.713 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.002405+0000", 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028507000000000001 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:786 3.1s0 3:8a412445:::smithi19530025-48:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.627999093, 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.113603414, 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.714 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 786 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.715 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991603+0000", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991604+0000", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4839999999999999e-06 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991627+0000", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2489000000000001e-05 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.716 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991645+0000", 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7861999999999999e-05 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991656+0000", 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0994e-05 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995156+0000", 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035008109999999999 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.717 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.995476+0000", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000319609 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.996848+0000", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0013722630000000001 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.000177+0000", 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0033282540000000001 2024-09-16T02:15:36.718 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.105020+0000", 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.104843271 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.105206+0000", 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018637699999999999 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:787 3.1s0 3:8a412445:::smithi19530025-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.627927312000001, 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.116105888, 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 787 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991675+0000", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991676+0000", 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.04e-06 2024-09-16T02:15:36.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991680+0000", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.2799999999999997e-06 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991692+0000", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1663e-05 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.991700+0000", 2024-09-16T02:15:36.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2160000000000006e-06 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.997760+0000", 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060602829999999996 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:18.998015+0000", 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025511099999999998 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.106099+0000", 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.10808414700000001 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.106254+0000", 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015439100000000001 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:19.107780+0000", 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015267569999999999 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.724 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:873 3.1s0 3:81c6c0a8:::smithi19530025-13:head [stat,delete] snapc 23=[23,22,21,20,1f,1e,1d,18,d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:26.922383+0000", 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 9.6972192990000003, 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.031164618000000002, 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:36.725 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 873 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922383+0000", 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922383+0000", 2024-09-16T02:15:36.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922386+0000", 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0769999999999999e-06 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922400+0000", 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4793999999999999e-05 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922433+0000", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.2224999999999997e-05 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.922459+0000", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6441999999999999e-05 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.927823+0000", 2024-09-16T02:15:36.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0053639350000000002 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.928088+0000", 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026457299999999999 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.930165+0000", 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0020773089999999998 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:36.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.936006+0000", 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.005841415 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.953365+0000", 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017358264000000002 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:26.953547+0000", 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00018258400000000001 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.730 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.731 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:36.731 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:36.731 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:36.733 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:36.734 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:36.732+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:36.834 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:36.832+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:36.905 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:36.905 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:36.905 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:36.905 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:36.919 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:36.920 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:36.926 INFO:tasks.rados.rados.0.smithi195.stdout:380: done (8 left) 2024-09-16T02:15:36.926 INFO:tasks.rados.rados.0.smithi195.stdout:381: snap_remove snap 28 2024-09-16T02:15:36.936 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:36.933+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:37.038 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:37.032+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:37.044 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:37.045 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:37.045 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:37.045 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:37.045 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:37.058 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:37.139 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:37.132+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:37.226 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:37.226 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:37.226 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:37.226 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.943348166999996, 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.227 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.228 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.229 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.230 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:15:37.231 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.232 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.943155391999994, 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:15:37.233 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.234 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.235 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:15:37.236 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.237 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:37.238 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.943097348999999, 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.239 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.240 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:15:37.241 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:15:37.242 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.243 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.913940418999999, 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:15:37.244 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.245 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:15:37.246 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:15:37.247 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.248 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:37.249 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.913887360000004, 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.250 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.251 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.252 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:15:37.253 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.254 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 72.878705897000003, 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.255 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:15:37.256 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.257 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:15:37.258 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.865493198999999, 2024-09-16T02:15:37.259 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:37.260 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.261 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.262 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:15:37.263 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:15:37.264 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.265 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 61.865383962000003, 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:15:37.266 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.267 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:15:37.268 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.269 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.270 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.271 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.316119780999998, 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.272 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.273 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.274 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:15:37.275 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:15:37.276 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.277 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.316004053999997, 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.278 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.279 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.280 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:15:37.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 56.294469270999997, 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.283 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:15:37.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 73/66 ECSubWrite(tid=66, reqid=osd.8.0:66, at_version=73'42, trim_to=0'0, roll_forward_to=72'38))", 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:37.287 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.267637225999998, 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018190284000000001, 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954947+0000", 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5069999999999999e-06 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954992+0000", 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4162000000000002e-05 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955233+0000", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000241566 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955251+0000", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7448999999999998e-05 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956483+0000", 2024-09-16T02:15:37.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012321509999999999 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956507+0000", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973035+0000", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016528547000000001 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973136+0000", 2024-09-16T02:15:37.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010087200000000001 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 73/32 ECSubWrite(tid=107, reqid=osd.5.0:107, at_version=73'75, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.263928685000003, 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016895095999999998, 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.292 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958656+0000", 2024-09-16T02:15:37.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5939999999999999e-06 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958707+0000", 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1008999999999999e-05 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958930+0000", 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022282899999999999 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958952+0000", 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2146000000000001e-05 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959126+0000", 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173807 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959156+0000", 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0022000000000002e-05 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975497+0000", 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016340924999999999 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975550+0000", 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2763999999999998e-05 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 35.258482430999997, 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014562088000000001, 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964103+0000", 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.26e-06 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964341+0000", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023783300000000001 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964452+0000", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110974 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964476+0000", 2024-09-16T02:15:37.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.393e-05 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964624+0000", 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014846400000000001 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964652+0000", 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7389000000000002e-05 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978619+0000", 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013967228 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978663+0000", 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.401e-05 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:37.301 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 29.156752400999999, 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79668697799999999, 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065832+0000", 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4780000000000001e-06 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065862+0000", 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9570999999999999e-05 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860567+0000", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79470507099999999 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860588+0000", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0665e-05 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860756+0000", 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016797700000000001 2024-09-16T02:15:37.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860787+0000", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1492000000000003e-05 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862431+0000", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016435880000000001 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862518+0000", 2024-09-16T02:15:37.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7136000000000001e-05 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=103, reqid=client.4598.0:770, at_version=87'86, trim_to=0'0, roll_forward_to=79'84))", 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 21.263879475, 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017445961999999999, 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958706+0000", 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1660000000000001e-06 2024-09-16T02:15:37.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.959284+0000", 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057834499999999997 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972561+0000", 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01327654 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972585+0000", 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4221999999999999e-05 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972741+0000", 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156248 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972764+0000", 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3136e-05 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976076+0000", 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003311858 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976150+0000", 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3447000000000003e-05 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=107, reqid=client.4598.0:774, at_version=87'90, trim_to=0'0, roll_forward_to=87'86))", 2024-09-16T02:15:37.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 21.237380921, 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19572739, 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985204+0000", 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2630000000000002e-06 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985253+0000", 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8288000000000003e-05 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179555+0000", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.194302157 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179570+0000", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5322999999999999e-05 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179746+0000", 2024-09-16T02:15:37.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175391 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179762+0000", 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6388999999999998e-05 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180848+0000", 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010859349999999999 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180930+0000", 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1644000000000006e-05 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:831 3.6s0 3:63499ffe:::smithi19530025-18:head [delete] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91)", 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.216744644999999, 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89186369399999998, 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 831 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0999999999999997e-07 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005846+0000", 2024-09-16T02:15:37.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1690000000000003e-06 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005911+0000", 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4146000000000006e-05 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005928+0000", 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7541999999999999e-05 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008320+0000", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002391476 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008367+0000", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.757e-05 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008731+0000", 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036390000000000001 2024-09-16T02:15:37.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.009626+0000", 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00089470299999999995 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897500+0000", 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88787382500000001 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897702+0000", 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020255300000000001 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.21253256, 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066496243999999996, 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.321 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.322 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010052+0000", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0899999999999999e-06 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010064+0000", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2726e-05 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076273+0000", 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066208820000000002 2024-09-16T02:15:37.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076290+0000", 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6654e-05 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076469+0000", 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178717 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076547+0000", 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8237000000000003e-05 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 17.211652902000001, 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88785698300000004, 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:37.325 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010932+0000", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.48e-06 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010939+0000", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897090+0000", 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88615089199999997 2024-09-16T02:15:37.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897108+0000", 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.77e-05 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897272+0000", 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016364099999999999 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:37.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.898787+0000", 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015155539999999999 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:37.329 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:37.331 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:37.331 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:37.326+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:37.432 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:37.430+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:37.463 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:37.464 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:37.464 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:37.464 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:37.477 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:15:37.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 11] out_osds: [10] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-16T02:15:37.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:37.487 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-16T02:15:37.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:15:37.533 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:37.530+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:37.594 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:37.595 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:37.595 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:37.595 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:37.595 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:37.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:15:37.634 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:37.634+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:37.727 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:37.727 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [] 2024-09-16T02:15:37.727 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:37.735 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:37.730+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:37.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:37.836 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:37.834+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:37.862 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:37.862 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:37.862 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:37.862 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:15:37.874 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:15:37.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:37.895 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-16T02:15:36.920604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112411187,"num_objects":88,"num_object_clones":37,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":5,"num_read":509,"num_read_kb":131669,"num_write":1017,"num_write_kb":155490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1361,"ondisk_log_size":1361,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":658248,"kb_used_data":202840,"kb_used_omap":23,"kb_used_meta":455016,"kb_avail":1414919352,"statfs":{"total":1449551462400,"available":1448877416448,"internally_reserved":0,"allocated":207708160,"data_stored":253325209,"data_compressed":3762574,"data_compressed_allocated":5173248,"data_compressed_original":11206656,"omap_allocated":23922,"internal_metadata":465937038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9158656,"num_objects":4,"num_object_clones":3,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":3073,"num_write":17,"num_write_kb":8944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.978407"},"pg_stats":[{"pgid":"3.f","version":"106'47","reported_seq":397,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.998790+0000","last_change":"2024-09-16T02:15:35.000286+0000","last_active":"2024-09-16T02:15:34.998790+0000","last_peered":"2024-09-16T02:15:34.998790+0000","last_clean":"2024-09-16T02:15:34.998790+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:15:34.998790+0000","last_undegraded":"2024-09-16T02:15:34.998790+0000","last_fullsized":"2024-09-16T02:15:34.998790+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:38.125773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027392118999999999,"stat_sum":{"num_bytes":4751360,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":2664,"num_write":18,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.e","version":"99'77","reported_seq":509,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-16T02:15:33.963453+0000","last_change":"2024-09-16T02:15:33.963963+0000","last_active":"2024-09-16T02:15:33.963453+0000","last_peered":"2024-09-16T02:15:33.963453+0000","last_clean":"2024-09-16T02:15:33.963453+0000","last_became_active":"2024-09-16T02:15:13.070271+0000","last_became_peered":"2024-09-16T02:15:13.070271+0000","last_unstale":"2024-09-16T02:15:33.963453+0000","last_undegraded":"2024-09-16T02:15:33.963453+0000","last_fullsized":"2024-09-16T02:15:33.963453+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'66","last_scrub_stamp":"2024-09-16T02:15:27.680026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.680026+0000","objects_scrubbed":8,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.196604+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00073733700000000002,"stat_sum":{"num_bytes":11616256,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":9623,"num_write":73,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,14,11,0,15],"acting":[8,9,1,14,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"3.d","version":"106'110","reported_seq":453,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.007990+0000","last_change":"2024-09-16T02:15:35.009149+0000","last_active":"2024-09-16T02:15:35.007990+0000","last_peered":"2024-09-16T02:15:35.007990+0000","last_clean":"2024-09-16T02:15:35.007990+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:15:35.007990+0000","last_undegraded":"2024-09-16T02:15:35.007990+0000","last_fullsized":"2024-09-16T02:15:35.007990+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:48.883671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041628440000000003,"stat_sum":{"num_bytes":5488640,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":15024,"num_write":58,"num_write_kb":12320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.c","version":"99'31","reported_seq":376,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-16T02:15:33.963109+0000","last_change":"2024-09-16T02:15:33.965851+0000","last_active":"2024-09-16T02:15:33.963109+0000","last_peered":"2024-09-16T02:15:33.963109+0000","last_clean":"2024-09-16T02:15:33.963109+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:15:33.963109+0000","last_undegraded":"2024-09-16T02:15:33.963109+0000","last_fullsized":"2024-09-16T02:15:33.963109+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'29","last_scrub_stamp":"2024-09-16T02:15:27.850068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.850068+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:28.279384+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0029702449999999998,"stat_sum":{"num_bytes":3162112,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7055,"num_write":13,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"3.b","version":"106'71","reported_seq":452,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.996041+0000","last_change":"2024-09-16T02:15:35.001752+0000","last_active":"2024-09-16T02:15:34.996041+0000","last_peered":"2024-09-16T02:15:34.996041+0000","last_clean":"2024-09-16T02:15:34.996041+0000","last_became_active":"2024-09-16T02:15:13.074136+0000","last_became_peered":"2024-09-16T02:15:13.074136+0000","last_unstale":"2024-09-16T02:15:34.996041+0000","last_undegraded":"2024-09-16T02:15:34.996041+0000","last_fullsized":"2024-09-16T02:15:34.996041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.249432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026872966000000002,"stat_sum":{"num_bytes":10207232,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":12153,"num_write":53,"num_write_kb":10416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8077312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,2,13],"acting":[7,15,1,9,8,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.968243+0000","last_change":"2024-09-16T02:15:34.969093+0000","last_active":"2024-09-16T02:15:34.968243+0000","last_peered":"2024-09-16T02:15:34.968243+0000","last_clean":"2024-09-16T02:15:34.968243+0000","last_became_active":"2024-09-16T02:15:13.070217+0000","last_became_peered":"2024-09-16T02:15:13.070217+0000","last_unstale":"2024-09-16T02:15:34.968243+0000","last_undegraded":"2024-09-16T02:15:34.968243+0000","last_fullsized":"2024-09-16T02:15:34.968243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001402737,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,14,1,9,8,15,2],"acting":[6,14,1,9,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.9","version":"106'28","reported_seq":287,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.990896+0000","last_change":"2024-09-16T02:15:34.993249+0000","last_active":"2024-09-16T02:15:34.990896+0000","last_peered":"2024-09-16T02:15:34.990896+0000","last_clean":"2024-09-16T02:15:34.990896+0000","last_became_active":"2024-09-16T02:14:43.948134+0000","last_became_peered":"2024-09-16T02:14:43.948134+0000","last_unstale":"2024-09-16T02:15:34.990896+0000","last_undegraded":"2024-09-16T02:15:34.990896+0000","last_fullsized":"2024-09-16T02:15:34.990896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'26","last_scrub_stamp":"2024-09-16T02:15:30.066659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:30.066659+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:37.191474+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.018542082000000001,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13,1,0,8,6,12],"acting":[7,13,1,0,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.8","version":"106'121","reported_seq":480,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T02:15:34.991958+0000","last_change":"2024-09-16T02:15:34.991958+0000","last_active":"2024-09-16T02:15:34.991958+0000","last_peered":"2024-09-16T02:15:34.991958+0000","last_clean":"2024-09-16T02:15:34.991958+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:15:34.991958+0000","last_undegraded":"2024-09-16T02:15:34.991958+0000","last_fullsized":"2024-09-16T02:15:34.991958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.428155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015878369,"stat_sum":{"num_bytes":9732096,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":18184,"num_write":67,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"1.0","version":"26'112","reported_seq":307,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.979268+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:15:34.979268+0000","last_peered":"2024-09-16T02:15:34.979268+0000","last_clean":"2024-09-16T02:15:34.979268+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:15:34.979268+0000","last_undegraded":"2024-09-16T02:15:34.979268+0000","last_fullsized":"2024-09-16T02:15:34.979268+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.992783+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:15:34.992783+0000","last_peered":"2024-09-16T02:15:34.992783+0000","last_clean":"2024-09-16T02:15:34.992783+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:15:34.992783+0000","last_undegraded":"2024-09-16T02:15:34.992783+0000","last_fullsized":"2024-09-16T02:15:34.992783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'157","reported_seq":565,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T02:15:35.000855+0000","last_change":"2024-09-16T02:15:34.980685+0000","last_active":"2024-09-16T02:15:35.000855+0000","last_peered":"2024-09-16T02:15:35.000855+0000","last_clean":"2024-09-16T02:15:35.000855+0000","last_became_active":"2024-09-16T02:15:13.072281+0000","last_became_peered":"2024-09-16T02:15:13.072281+0000","last_unstale":"2024-09-16T02:15:35.000855+0000","last_undegraded":"2024-09-16T02:15:35.000855+0000","last_fullsized":"2024-09-16T02:15:35.000855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:44.382777+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017367517999999998,"stat_sum":{"num_bytes":17940480,"num_objects":13,"num_object_clones":6,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":68,"num_read_kb":17164,"num_write":128,"num_write_kb":21536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,4,14,1,0],"acting":[13,12,6,4,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.971989+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:15:34.971989+0000","last_peered":"2024-09-16T02:15:34.971989+0000","last_clean":"2024-09-16T02:15:34.971989+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:15:34.971989+0000","last_undegraded":"2024-09-16T02:15:34.971989+0000","last_fullsized":"2024-09-16T02:15:34.971989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:25.996964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"106'228","reported_seq":623,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.059109+0000","last_change":"2024-09-16T02:15:35.060305+0000","last_active":"2024-09-16T02:15:35.059109+0000","last_peered":"2024-09-16T02:15:35.059109+0000","last_clean":"2024-09-16T02:15:35.059109+0000","last_became_active":"2024-09-16T02:14:58.940506+0000","last_became_peered":"2024-09-16T02:14:58.940506+0000","last_unstale":"2024-09-16T02:15:35.059109+0000","last_undegraded":"2024-09-16T02:15:35.059109+0000","last_fullsized":"2024-09-16T02:15:35.059109+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'209","last_scrub_stamp":"2024-09-16T02:15:29.441137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:29.441137+0000","objects_scrubbed":13,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:29.660064+0000","scrub_duration":23,"objects_trimmed":1,"snaptrim_duration":0.073047173000000007,"stat_sum":{"num_bytes":18038784,"num_objects":12,"num_object_clones":6,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":53,"num_read_kb":17941,"num_write":186,"num_write_kb":24992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.979406+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:15:34.979406+0000","last_peered":"2024-09-16T02:15:34.979406+0000","last_clean":"2024-09-16T02:15:34.979406+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:15:34.979406+0000","last_undegraded":"2024-09-16T02:15:34.979406+0000","last_fullsized":"2024-09-16T02:15:34.979406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:20.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"48'8","reported_seq":351,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.967490+0000","last_change":"2024-09-16T02:15:34.968351+0000","last_active":"2024-09-16T02:15:34.967490+0000","last_peered":"2024-09-16T02:15:34.967490+0000","last_clean":"2024-09-16T02:15:34.967490+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:15:34.967490+0000","last_undegraded":"2024-09-16T02:15:34.967490+0000","last_fullsized":"2024-09-16T02:15:34.967490+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013024899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.972341+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:15:34.972341+0000","last_peered":"2024-09-16T02:15:34.972341+0000","last_clean":"2024-09-16T02:15:34.972341+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:15:34.972341+0000","last_undegraded":"2024-09-16T02:15:34.972341+0000","last_fullsized":"2024-09-16T02:15:34.972341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"106'29","reported_seq":438,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.999643+0000","last_change":"2024-09-16T02:15:35.001733+0000","last_active":"2024-09-16T02:15:34.999643+0000","last_peered":"2024-09-16T02:15:34.999643+0000","last_clean":"2024-09-16T02:15:34.999643+0000","last_became_active":"2024-09-16T02:15:13.074572+0000","last_became_peered":"2024-09-16T02:15:13.074572+0000","last_unstale":"2024-09-16T02:15:34.999643+0000","last_undegraded":"2024-09-16T02:15:34.999643+0000","last_fullsized":"2024-09-16T02:15:34.999643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021639180000000001,"stat_sum":{"num_bytes":1687552,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":5328,"num_write":29,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,12],"acting":[13,2,11,14,4,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.5","version":"106'66","reported_seq":155,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.985612+0000","last_change":"2024-09-16T02:15:34.987430+0000","last_active":"2024-09-16T02:15:34.985612+0000","last_peered":"2024-09-16T02:15:34.985612+0000","last_clean":"2024-09-16T02:15:34.985612+0000","last_became_active":"2024-09-16T02:15:13.073157+0000","last_became_peered":"2024-09-16T02:15:13.073157+0000","last_unstale":"2024-09-16T02:15:34.985612+0000","last_undegraded":"2024-09-16T02:15:34.985612+0000","last_fullsized":"2024-09-16T02:15:34.985612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020256013999999999,"stat_sum":{"num_bytes":8912896,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":21195,"num_write":34,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5144576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,15,14,0,7,8],"acting":[12,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.987473+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:15:34.987473+0000","last_peered":"2024-09-16T02:15:34.987473+0000","last_clean":"2024-09-16T02:15:34.987473+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:15:34.987473+0000","last_undegraded":"2024-09-16T02:15:34.987473+0000","last_fullsized":"2024-09-16T02:15:34.987473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"106'98","reported_seq":478,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.018843+0000","last_change":"2024-09-16T02:15:35.019924+0000","last_active":"2024-09-16T02:15:35.018843+0000","last_peered":"2024-09-16T02:15:35.018843+0000","last_clean":"2024-09-16T02:15:35.018843+0000","last_became_active":"2024-09-16T02:14:55.933032+0000","last_became_peered":"2024-09-16T02:14:55.933032+0000","last_unstale":"2024-09-16T02:15:35.018843+0000","last_undegraded":"2024-09-16T02:15:35.018843+0000","last_fullsized":"2024-09-16T02:15:35.018843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:41.156017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043122655000000003,"stat_sum":{"num_bytes":4014080,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":1359,"num_write":93,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4112384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":172,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-16T02:15:33.962900+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:15:33.962900+0000","last_peered":"2024-09-16T02:15:33.962900+0000","last_clean":"2024-09-16T02:15:33.962900+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:15:33.962900+0000","last_undegraded":"2024-09-16T02:15:33.962900+0000","last_fullsized":"2024-09-16T02:15:33.962900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:21.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"106'134","reported_seq":470,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.004474+0000","last_change":"2024-09-16T02:15:35.004474+0000","last_active":"2024-09-16T02:15:35.004474+0000","last_peered":"2024-09-16T02:15:35.004474+0000","last_clean":"2024-09-16T02:15:35.004474+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:15:35.004474+0000","last_undegraded":"2024-09-16T02:15:35.004474+0000","last_fullsized":"2024-09-16T02:15:35.004474+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026077304999999999,"stat_sum":{"num_bytes":9994240,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":2074,"num_write":82,"num_write_kb":13776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-16T02:15:33.957978+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:15:33.957978+0000","last_peered":"2024-09-16T02:15:33.957978+0000","last_clean":"2024-09-16T02:15:33.957978+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:15:33.957978+0000","last_undegraded":"2024-09-16T02:15:33.957978+0000","last_fullsized":"2024-09-16T02:15:33.957978+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"99'41","reported_seq":385,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-16T02:15:33.958289+0000","last_change":"2024-09-16T02:15:33.959923+0000","last_active":"2024-09-16T02:15:33.958289+0000","last_peered":"2024-09-16T02:15:33.958289+0000","last_clean":"2024-09-16T02:15:33.958289+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:15:33.958289+0000","last_undegraded":"2024-09-16T02:15:33.958289+0000","last_fullsized":"2024-09-16T02:15:33.958289+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024157279999999998,"stat_sum":{"num_bytes":2506752,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":1828,"num_write":16,"num_write_kb":2736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.967835+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:15:34.967835+0000","last_peered":"2024-09-16T02:15:34.967835+0000","last_clean":"2024-09-16T02:15:34.967835+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:15:34.967835+0000","last_undegraded":"2024-09-16T02:15:34.967835+0000","last_fullsized":"2024-09-16T02:15:34.967835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111820800,"num_objects":84,"num_object_clones":37,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":5,"num_read":423,"num_read_kb":131596,"num_write":870,"num_write_kb":152816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":199172096,"data_stored":247599104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},"log_size":1246,"ondisk_log_size":1246,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47560,"kb_used_data":12208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324280,"statfs":{"total":96636764160,"available":96588062720,"internally_reserved":0,"allocated":12500992,"data_stored":17089001,"data_compressed":279437,"data_compressed_allocated":380928,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44528,"kb_used_data":13332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327312,"statfs":{"total":96636764160,"available":96591167488,"internally_reserved":0,"allocated":13651968,"data_stored":17502153,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280534,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41012,"kb_used_data":9824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330828,"statfs":{"total":96636764160,"available":96594767872,"internally_reserved":0,"allocated":10059776,"data_stored":12771273,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313238,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48900,"kb_used_data":17708,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94322940,"statfs":{"total":96636764160,"available":96586690560,"internally_reserved":0,"allocated":18132992,"data_stored":21591807,"data_compressed":243868,"data_compressed_allocated":335872,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46984,"kb_used_data":15788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324856,"statfs":{"total":96636764160,"available":96588652544,"internally_reserved":0,"allocated":16166912,"data_stored":20910025,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313236,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47504,"kb_used_data":16312,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324336,"statfs":{"total":96636764160,"available":96588120064,"internally_reserved":0,"allocated":16703488,"data_stored":20248319,"data_compressed":243868,"data_compressed_allocated":335872,"data_compressed_original":671744,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41772,"kb_used_data":10580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330068,"statfs":{"total":96636764160,"available":96593989632,"internally_reserved":0,"allocated":10833920,"data_stored":13400230,"data_compressed":249942,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36968,"kb_used_data":9932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334872,"statfs":{"total":96636764160,"available":96598908928,"internally_reserved":0,"allocated":10170368,"data_stored":11517897,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46156,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325684,"statfs":{"total":96636764160,"available":96589500416,"internally_reserved":0,"allocated":15319040,"data_stored":18751452,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280533,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38792,"kb_used_data":11764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94333048,"statfs":{"total":96636764160,"available":96597041152,"internally_reserved":0,"allocated":12046336,"data_stored":13842175,"data_compressed":243868,"data_compressed_allocated":335872,"data_compressed_original":671744,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50876,"kb_used_data":19684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320964,"statfs":{"total":96636764160,"available":96584667136,"internally_reserved":0,"allocated":20156416,"data_stored":24588777,"data_compressed":279437,"data_compressed_allocated":380928,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43788,"kb_used_data":12592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328052,"statfs":{"total":96636764160,"available":96591925248,"internally_reserved":0,"allocated":12894208,"data_stored":14581705,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37324,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334516,"statfs":{"total":96636764160,"available":96598544384,"internally_reserved":0,"allocated":10539008,"data_stored":12214217,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45968,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325872,"statfs":{"total":96636764160,"available":96589692928,"internally_reserved":0,"allocated":15134720,"data_stored":18501577,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40116,"kb_used_data":13084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331724,"statfs":{"total":96636764160,"available":96595685376,"internally_reserved":0,"allocated":13398016,"data_stored":15814601,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":0,"seq":429496729602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18874368,"data_stored":22790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14946304,"data_stored":19677184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9596928,"data_stored":12472320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":14692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9318400,"data_stored":11005952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9863168,"data_stored":11628544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13914112,"data_stored":17207296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15007744,"data_stored":18923520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":10838016,"data_stored":12935168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12177408,"data_stored":14811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7139328,"data_stored":8015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8839168,"data_stored":11792384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12431360,"data_stored":16293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11218944,"data_stored":15327232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":16924672,"data_stored":20684800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":15495168,"data_stored":19341312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:15:37.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 16 2024-09-16T02:15:37.927 INFO:tasks.rados.rados.0.smithi195.stdout:381: done (8 left) 2024-09-16T02:15:37.927 INFO:tasks.rados.rados.0.smithi195.stdout:382: delete oid 10 current snap is 37 2024-09-16T02:15:37.927 INFO:tasks.rados.rados.0.smithi195.stdout:371: finishing copy_from to smithi19530025-42 2024-09-16T02:15:37.927 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 42 v 164 (ObjNum 123 snap 27 seq_num 123) dirty exists 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:377: finishing copy_from to smithi19530025-37 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:377: got expected ENOENT (src dne) 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:377: finishing copy_from racing read to smithi19530025-37 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:374: finishing rollback tid 0 to smithi19530025-23 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:374: finishing rollback tid 1 to smithi19530025-23 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:374: finishing rollback tid 2 to smithi19530025-23 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 23 v 167 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 12 v 49 (ObjNum 136 snap 33 seq_num 136) dirty exists 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:379: finishing rollback tid 0 to smithi19530025-8 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:379: finishing rollback tid 1 to smithi19530025-8 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:373: finishing rollback tid 0 to smithi19530025-9 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:379: finishing rollback tid 2 to smithi19530025-8 2024-09-16T02:15:37.928 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 8 v 70 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-16T02:15:37.929 INFO:tasks.rados.rados.0.smithi195.stdout:373: finishing rollback tid 1 to smithi19530025-9 2024-09-16T02:15:37.929 INFO:tasks.rados.rados.0.smithi195.stdout:375: expect (ObjNum 124 snap 27 seq_num 124) 2024-09-16T02:15:37.936 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:37.934+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:37.941 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1709151 2024-09-16T02:15:37.970 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 1855964 2024-09-16T02:15:37.982 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 386394 2024-09-16T02:15:38.002 INFO:tasks.rados.rados.0.smithi195.stdout:373: finishing rollback tid 2 to smithi19530025-9 2024-09-16T02:15:38.002 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 9 v 118 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-16T02:15:38.002 INFO:tasks.rados.rados.0.smithi195.stdout:378: expect (ObjNum 75 snap 8 seq_num 75) 2024-09-16T02:15:38.002 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:38.013 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:38.019 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-09-16T02:15:38.019 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-09-16T02:15:38.019 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:38.023 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:38.035 INFO:tasks.rados.rados.0.smithi195.stdout:371: done (8 left) 2024-09-16T02:15:38.035 INFO:tasks.rados.rados.0.smithi195.stdout:373: done (7 left) 2024-09-16T02:15:38.035 INFO:tasks.rados.rados.0.smithi195.stdout:374: done (6 left) 2024-09-16T02:15:38.035 INFO:tasks.rados.rados.0.smithi195.stdout:375: done (5 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:376: done (4 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:377: done (3 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:378: done (2 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:379: done (1 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:382: done (0 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:383: read oid 12 snap 21 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:383: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:384: rmattr oid 29 current snap is 37 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:384: done (1 left) 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 36 current snap is 37 2024-09-16T02:15:38.036 INFO:tasks.rados.rados.0.smithi195.stdout:385: seq_num 147 ranges {0=65536} 2024-09-16T02:15:38.038 INFO:tasks.rados.rados.0.smithi195.stdout:385: writing smithi19530025-36 from 0 to 65536 tid 1 2024-09-16T02:15:38.038 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 29 current snap is 37 2024-09-16T02:15:38.038 INFO:tasks.rados.rados.0.smithi195.stdout:386: seq_num 148 ranges {2457600=81920} 2024-09-16T02:15:38.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:38.039 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:38.032+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:38.077 INFO:tasks.rados.rados.0.smithi195.stdout:386: writing smithi19530025-29 from 2457600 to 2539520 tid 1 2024-09-16T02:15:38.077 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 44 current snap is 37 2024-09-16T02:15:38.077 INFO:tasks.rados.rados.0.smithi195.stdout:387: seq_num 149 ranges {2375680=98304} 2024-09-16T02:15:38.105 INFO:tasks.rados.rados.0.smithi195.stdout:387: writing smithi19530025-44 from 2375680 to 2473984 tid 1 2024-09-16T02:15:38.105 INFO:tasks.rados.rados.0.smithi195.stdout:388: setattr oid 6 current snap is 37 2024-09-16T02:15:38.105 INFO:tasks.rados.rados.0.smithi195.stdout:383: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-16T02:15:38.109 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 195532 2024-09-16T02:15:38.111 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 148101 2024-09-16T02:15:38.113 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:385: finishing write tid 1 to smithi19530025-36 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:385: finishing write tid 2 to smithi19530025-36 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:385: finishing write tid 3 to smithi19530025-36 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:385: oid 36 updating version 0 to 167 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:385: oid 36 version 167 is already newer than 166 2024-09-16T02:15:38.121 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 36 v 167 (ObjNum 147 snap 37 seq_num 147) dirty exists 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:385: left oid 36 (ObjNum 147 snap 37 seq_num 147) 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: finishing write tid 1 to smithi19530025-29 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: finishing write tid 2 to smithi19530025-29 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: finishing write tid 3 to smithi19530025-29 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: oid 29 updating version 0 to 78 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: oid 29 version 78 is already newer than 77 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 29 v 78 (ObjNum 148 snap 37 seq_num 148) dirty exists 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:386: left oid 29 (ObjNum 148 snap 37 seq_num 148) 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:387: finishing write tid 1 to smithi19530025-44 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:387: finishing write tid 2 to smithi19530025-44 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:387: finishing write tid 3 to smithi19530025-44 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:387: oid 44 updating version 0 to 236 2024-09-16T02:15:38.122 INFO:tasks.rados.rados.0.smithi195.stdout:387: oid 44 updating version 236 to 237 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 44 v 237 (ObjNum 149 snap 37 seq_num 149) dirty exists 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:387: left oid 44 (ObjNum 149 snap 37 seq_num 149) 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:383: done (4 left) 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:385: done (3 left) 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:386: done (2 left) 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:387: done (1 left) 2024-09-16T02:15:38.123 INFO:tasks.rados.rados.0.smithi195.stdout:389: delete oid 46 current snap is 37 2024-09-16T02:15:38.125 INFO:tasks.rados.rados.0.smithi195.stdout:389: done (1 left) 2024-09-16T02:15:38.125 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 30 current snap is 37 2024-09-16T02:15:38.125 INFO:tasks.rados.rados.0.smithi195.stdout:390: seq_num 151 ranges {2457600=131072} 2024-09-16T02:15:38.138 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:38.138+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:38.157 INFO:tasks.rados.rados.0.smithi195.stdout:390: writing smithi19530025-30 from 2457600 to 2588672 tid 1 2024-09-16T02:15:38.157 INFO:tasks.rados.rados.0.smithi195.stdout:391: snap_remove snap 35 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.4215.0:124 1.0 e24/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 24'94, mlcod=24'89)", 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:38.166 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 91.239309689999999, 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33740482100000002, 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924420+0000", 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924422+0000", 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2929999999999999e-06 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:06.924431+0000", 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8769999999999999e-06 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260013+0000", 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.33558222100000001 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260031+0000", 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8063e-05 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260228+0000", 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000196782 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.260273+0000", 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5877000000000001e-05 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261745+0000", 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001471916 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:07.261825+0000", 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9791999999999998e-05 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.171 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 45/32 ECSubWrite(tid=52, reqid=osd.11.0:52, at_version=45'44, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 63.801651702000001, 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080847929999999998, 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362078+0000", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362079+0000", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.054e-06 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362106+0000", 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6322e-05 2024-09-16T02:15:38.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362152+0000", 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6276999999999998e-05 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362163+0000", 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.064e-05 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362225+0000", 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2539999999999994e-05 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.362236+0000", 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1151999999999999e-05 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370090+0000", 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00785368 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:34.370163+0000", 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3127999999999995e-05 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 66/32 ECSubWrite(tid=73, reqid=client.4598.0:580, at_version=66'64, trim_to=0'0, roll_forward_to=66'59))", 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 43.114214898999997, 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51414269899999998, 2024-09-16T02:15:38.176 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049515+0000", 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049516+0000", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.68e-07 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.049529+0000", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2897e-05 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562582+0000", 2024-09-16T02:15:38.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.51305329300000002 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562593+0000", 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1092e-05 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562676+0000", 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2935000000000005e-05 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.562686+0000", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0067e-05 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563613+0000", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092661900000000005 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:14:55.563658+0000", 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5127999999999999e-05 2024-09-16T02:15:38.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 36.210959885999998, 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073114410000000001, 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952770+0000", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952772+0000", 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3300000000000001e-06 2024-09-16T02:15:38.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952824+0000", 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1836000000000003e-05 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952926+0000", 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101223 2024-09-16T02:15:38.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.952948+0000", 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2478999999999999e-05 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953117+0000", 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016898299999999999 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.953144+0000", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7036e-05 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.959979+0000", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068347410000000001 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:01.960082+0000", 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010281300000000001 2024-09-16T02:15:38.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 73/32 ECSubWrite(tid=274, reqid=osd.13.0:274, at_version=73'84, trim_to=0'0, roll_forward_to=73'82))", 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 35.849695294, 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047083613000000003, 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314035+0000", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314037+0000", 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7940000000000001e-06 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.314080+0000", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3053000000000003e-05 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359609+0000", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045529636999999998 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359632+0000", 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2869000000000001e-05 2024-09-16T02:15:38.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359778+0000", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014631299999999999 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.359798+0000", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9099000000000001e-05 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361025+0000", 2024-09-16T02:15:38.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012279420000000001 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.361118+0000", 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2905999999999994e-05 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:38.190 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 35.204892678, 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010173534, 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958837+0000", 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958839+0000", 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7570000000000001e-06 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958892+0000", 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2469999999999997e-05 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.958988+0000", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6779000000000003e-05 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.959011+0000", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.245e-05 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961665+0000", 2024-09-16T02:15:38.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026538410000000001 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961690+0000", 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5313e-05 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.968914+0000", 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072245180000000001 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6405999999999998e-05 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 74/32 ECSubWrite(tid=111, reqid=client.4598.0:654, at_version=74'81, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 35.202268853, 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011431721000000001, 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.195 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961461+0000", 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961463+0000", 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.68e-06 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961515+0000", 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2586000000000001e-05 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961613+0000", 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7598999999999998e-05 2024-09-16T02:15:38.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.961635+0000", 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1780000000000002e-05 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962881+0000", 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012456850000000001 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.962897+0000", 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6303e-05 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972785+0000", 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098886259999999993 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:02.972893+0000", 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107462 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.199 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpRead(3.2s6 84/83 ECSubRead(tid=314, to_read={3:476b4430:::smithi19530025-30:14=262144,262144,0}, subchunks={3:476b4430:::smithi19530025-30:14=[0,1]}, attrs_to_read=))", 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.934658461000001, 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57667217800000004, 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229072+0000", 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-09-16T02:15:38.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.229083+0000", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0162e-05 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.804968+0000", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.57588503999999996 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805000+0000", 2024-09-16T02:15:38.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2283000000000002e-05 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805176+0000", 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017626499999999999 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:13.805744+0000", 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056751299999999996 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=160, reqid=client.4598.0:758, at_version=85'151, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.081148470999999, 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092037909999999994, 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082582+0000", 2024-09-16T02:15:38.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000001e-07 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082613+0000", 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0790000000000002e-05 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082884+0000", 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027123099999999998 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082902+0000", 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.753e-05 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085344+0000", 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024427120000000001 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.085371+0000", 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6587000000000001e-05 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091675+0000", 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063039510000000003 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.091785+0000", 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110436 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.208 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 85/69 ECSubWrite(tid=161, reqid=client.4598.0:759, at_version=85'152, trim_to=0'0, roll_forward_to=79'148))", 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.081112966999999, 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010507528, 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082617+0000", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082618+0000", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0999999999999999e-07 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.082651+0000", 2024-09-16T02:15:38.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3246e-05 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083064+0000", 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041363000000000001 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.083085+0000", 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.048e-05 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089109+0000", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060238330000000001 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.089137+0000", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8569999999999999e-05 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093029+0000", 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038919520000000002 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.093125+0000", 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5307000000000001e-05 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 85/83 ECSubWrite(tid=320, reqid=client.4598.0:756, at_version=85'108, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.071441802999999, 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076015120000000004, 2024-09-16T02:15:38.213 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092288+0000", 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092289+0000", 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9599999999999998e-07 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.092436+0000", 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000146983 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096144+0000", 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037081829999999999 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096168+0000", 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3782999999999999e-05 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096349+0000", 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018129600000000001 2024-09-16T02:15:38.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096382+0000", 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2252000000000002e-05 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099809+0000", 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034276129999999999 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.099890+0000", 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0605999999999994e-05 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.2s6 85/83 ECSubWrite(tid=321, reqid=osd.13.0:321, at_version=85'110, trim_to=0'0, roll_forward_to=84'106))", 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.067765866999999, 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96998350899999997, 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.218 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095964+0000", 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.095966+0000", 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0540000000000002e-06 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:14.096092+0000", 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012574600000000001 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064646+0000", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96855418100000001 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064663+0000", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.645e-05 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064759+0000", 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6565999999999997e-05 2024-09-16T02:15:38.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.064772+0000", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2398000000000001e-05 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.065889+0000", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011169979999999999 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:15.065948+0000", 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9116000000000001e-05 2024-09-16T02:15:38.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 90/69 ECSubWrite(tid=173, reqid=client.4598.0:786, at_version=90'164, trim_to=0'0, roll_forward_to=90'161))", 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 19.166499583, 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107580065, 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997230+0000", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997231+0000", 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9899999999999999e-07 2024-09-16T02:15:38.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:18.997248+0000", 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6906e-05 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.102924+0000", 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10567633799999999 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.102943+0000", 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8827000000000001e-05 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.103113+0000", 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169727 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.103131+0000", 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7847e-05 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.104732+0000", 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016006919999999999 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.104811+0000", 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8929000000000003e-05 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.1s5 91/69 ECSubWrite(tid=178, reqid=osd.11.0:178, at_version=91'170, trim_to=0'0, roll_forward_to=90'164))", 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.199050262, 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007284876, 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964680+0000", 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964682+0000", 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2369999999999999e-06 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964741+0000", 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8904999999999999e-05 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964843+0000", 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010174 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.964868+0000", 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5795999999999998e-05 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.965026+0000", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015726600000000001 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.965052+0000", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5930000000000001e-05 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.971919+0000", 2024-09-16T02:15:38.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006867153 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:19.971965+0000", 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5849000000000001e-05 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.5s4 91/83 ECSubWrite(tid=43, reqid=client.4598.0:838, at_version=91'58, trim_to=0'0, roll_forward_to=90'56))", 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:38.232 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 17.217569837999999, 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083676029999999995, 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946160+0000", 2024-09-16T02:15:38.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.946161+0000", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.113e-06 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948556+0000", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023948060000000002 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948730+0000", 2024-09-16T02:15:38.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000174105 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948768+0000", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7423999999999999e-05 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948883+0000", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011533599999999999 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.948894+0000", 2024-09-16T02:15:38.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.136e-05 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.954390+0000", 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.005495977 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:20.954528+0000", 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137482 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 107/32 ECSubWrite(tid=130, reqid=client.4598.0:949, at_version=107'117, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:35.936266+0000", 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.2274640510000001, 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012795984, 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936266+0000", 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936266+0000", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936268+0000", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0930000000000001e-06 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936318+0000", 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9679000000000003e-05 2024-09-16T02:15:38.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936818+0000", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049984699999999999 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936844+0000", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6033e-05 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936939+0000", 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5480000000000001e-05 2024-09-16T02:15:38.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.936952+0000", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2622e-05 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.948990+0000", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012038040999999999 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.949062+0000", 2024-09-16T02:15:38.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2188999999999996e-05 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 107/32 ECSubWrite(tid=131, reqid=client.4598.0:950, at_version=107'118, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:35.938172+0000", 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 2.225558215, 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01093205, 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.241 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938172+0000", 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938172+0000", 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938173+0000", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.46e-06 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938215+0000", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1436000000000001e-05 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938302+0000", 2024-09-16T02:15:38.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7403000000000003e-05 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938324+0000", 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1916000000000002e-05 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938513+0000", 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000188569 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.938538+0000", 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5735999999999999e-05 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.949069+0000", 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010530217 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:35.949104+0000", 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5312999999999999e-05 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.245 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDScrubReserve(3.2s6,QREQUEST,e:108,from:13(0),reservation_nonce:2)", 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:37.112468+0000", 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 1.0512618010000001, 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12399007099999999, 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.246 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.112468+0000", 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.112468+0000", 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.112470+0000", 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.86e-06 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.247 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.112480+0000", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0283e-05 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.236170+0000", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12368965 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.236185+0000", 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4766999999999999e-05 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.248 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.236340+0000", 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015513899999999999 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.236384+0000", 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4415000000000001e-05 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.249 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.236458+0000", 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3956999999999997e-05 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.ds2 109/32 ECSubWrite(tid=136, reqid=osd.6.0:136, at_version=109'124, trim_to=0'0, roll_forward_to=108'122))", 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:37.941675+0000", 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.222054999, 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090317230000000002, 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.250 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941675+0000", 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941675+0000", 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941675+0000", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5800000000000001e-07 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941695+0000", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9881000000000002e-05 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941730+0000", 2024-09-16T02:15:38.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5058000000000002e-05 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941736+0000", 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0580000000000003e-06 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941800+0000", 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3510999999999996e-05 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.941811+0000", 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1246e-05 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.950664+0000", 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0088526160000000007 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.950707+0000", 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3195000000000002e-05 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.254 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDECSubOpWrite(3.8s2 109/32 ECSubWrite(tid=162, reqid=osd.5.0:162, at_version=109'123, trim_to=0'0, roll_forward_to=106'121))", 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-09-16T02:15:37.943057+0000", 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.22067287599999999, 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079991820000000005, 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "continuous": false, 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-09-16T02:15:38.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943057+0000", 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943057+0000", 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943058+0000", 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0299999999999998e-07 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943077+0000", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8641e-05 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943103+0000", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6519000000000001e-05 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943109+0000", 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6099999999999997e-06 2024-09-16T02:15:38.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943166+0000", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7027999999999997e-05 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_started", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.943174+0000", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9659999999999996e-06 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_committed", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.951032+0000", 2024-09-16T02:15:38.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078584009999999992 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-09-16T02:15:37.951056+0000", 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4314000000000001e-05 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-09-16T02:15:38.259 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:38.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:38.261 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:38.260+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:38.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:38.364 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:38.358+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:38.441 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:15:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-09-16T02:15:38.442 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-09-16T02:15:38.442 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:15:38.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:38.462 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:38.458+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:38.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:38.464 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:38.489 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:38.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:38.515 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:38.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:38.541 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:38.541 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:38.562 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:38.558+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:38.563 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:38.563 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:38.579 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:38.579 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:38.589 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:38.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:38.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:38.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:38.618 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:38.646 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:38.647 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:38.663 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:38.661+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:38.671 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:38.702 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:38.702 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:38.731 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:38.731 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:38.736 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:38.736 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:38.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:38.764 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:38.764 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:38.764 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:38.762+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:38.790 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:38.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:38.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:38.821 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:38.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:38.826+0000 7f735ffff640 0 -- 172.21.15.3:0/1007672791 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7f7368222330 msgr2=0x7f7368224720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:38.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:38.859 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:38.866 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:38.862+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:38.889 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:38.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:38.900 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:38.900 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:38.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:38.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:38.966 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:38.962+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:38.980 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 3 pgp_num to 16 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:391: done (2 left) 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 41 current snap is 37 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 6 v 47 (ObjNum 128 snap 27 seq_num 128) dirty exists 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: finishing write tid 1 to smithi19530025-30 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: finishing write tid 2 to smithi19530025-30 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: finishing write tid 3 to smithi19530025-30 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: oid 30 updating version 0 to 171 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: oid 30 updating version 171 to 172 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 30 v 172 (ObjNum 151 snap 37 seq_num 151) dirty exists 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:390: left oid 30 (ObjNum 151 snap 37 seq_num 151) 2024-09-16T02:15:38.989 INFO:tasks.rados.rados.0.smithi195.stdout:392: seq_num 152 ranges {0=147456} 2024-09-16T02:15:38.990 INFO:tasks.rados.rados.0.smithi195.stdout:392: writing smithi19530025-41 from 0 to 147456 tid 1 2024-09-16T02:15:38.990 INFO:tasks.rados.rados.0.smithi195.stdout:388: done (2 left) 2024-09-16T02:15:38.990 INFO:tasks.rados.rados.0.smithi195.stdout:390: done (1 left) 2024-09-16T02:15:38.990 INFO:tasks.rados.rados.0.smithi195.stdout:393: delete oid 34 current snap is 37 2024-09-16T02:15:39.017 INFO:tasks.rados.rados.0.smithi195.stdout:392: finishing write tid 1 to smithi19530025-41 2024-09-16T02:15:39.018 INFO:tasks.rados.rados.0.smithi195.stdout:392: finishing write tid 2 to smithi19530025-41 2024-09-16T02:15:39.018 INFO:tasks.rados.rados.0.smithi195.stdout:393: done (1 left) 2024-09-16T02:15:39.019 INFO:tasks.rados.rados.0.smithi195.stdout:394: read oid 50 snap 29 2024-09-16T02:15:39.019 INFO:tasks.rados.rados.0.smithi195.stdout:394: expect (ObjNum 127 snap 27 seq_num 127) 2024-09-16T02:15:39.019 INFO:tasks.rados.rados.0.smithi195.stdout:append oid 5 current snap is 37 2024-09-16T02:15:39.019 INFO:tasks.rados.rados.0.smithi195.stdout:395: seq_num 154 ranges {1507328=180224} 2024-09-16T02:15:39.045 INFO:tasks.rados.rados.0.smithi195.stdout:395: writing smithi19530025-5 from 1507328 to 1687552 tid 1 2024-09-16T02:15:39.045 INFO:tasks.rados.rados.0.smithi195.stdout:396: delete oid 4 current snap is 37 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:392: finishing write tid 3 to smithi19530025-41 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:392: oid 41 updating version 0 to 238 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:392: oid 41 updating version 238 to 239 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 41 v 239 (ObjNum 152 snap 37 seq_num 152) dirty exists 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:392: left oid 41 (ObjNum 152 snap 37 seq_num 152) 2024-09-16T02:15:39.046 INFO:tasks.rados.rados.0.smithi195.stdout:394: expect (ObjNum 127 snap 27 seq_num 127) 2024-09-16T02:15:39.067 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:39.066+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:39.090 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 3225214 2024-09-16T02:15:39.104 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:39.104 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.105 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:39.160 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 167014 2024-09-16T02:15:39.167 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:39.166+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:39.210 INFO:tasks.rados.rados.0.smithi195.stdout:395: finishing write tid 1 to smithi19530025-5 2024-09-16T02:15:39.269 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:39.268+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:39.278 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.279 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:39.370 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:39.364+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:39.418 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.418 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:39.471 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:39.469+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:39.559 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.559 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.571 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:39.570+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:39.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:39.591 INFO:tasks.rados.rados.0.smithi195.stdout:395: finishing write tid 2 to smithi19530025-5 2024-09-16T02:15:39.593 INFO:tasks.rados.rados.0.smithi195.stdout:395: finishing write tid 3 to smithi19530025-5 2024-09-16T02:15:39.593 INFO:tasks.rados.rados.0.smithi195.stdout:395: oid 5 updating version 0 to 79 2024-09-16T02:15:39.593 INFO:tasks.rados.rados.0.smithi195.stdout:395: oid 5 version 79 is already newer than 78 2024-09-16T02:15:39.593 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 5 v 79 (ObjNum 154 snap 37 seq_num 154) dirty exists 2024-09-16T02:15:39.593 INFO:tasks.rados.rados.0.smithi195.stdout:395: left oid 5 (ObjNum 154 snap 37 seq_num 154) 2024-09-16T02:15:39.673 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:39.668+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:39.736 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.736 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:39.773 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:39.770+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:39.864 INFO:tasks.rados.rados.0.smithi195.stdout:392: done (3 left) 2024-09-16T02:15:39.864 INFO:tasks.rados.rados.0.smithi195.stdout:394: done (2 left) 2024-09-16T02:15:39.864 INFO:tasks.rados.rados.0.smithi195.stdout:395: done (1 left) 2024-09-16T02:15:39.864 INFO:tasks.rados.rados.0.smithi195.stdout:396: done (0 left) 2024-09-16T02:15:39.864 INFO:tasks.rados.rados.0.smithi195.stdout:397: rmattr oid 25 current snap is 37 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout:397: done (0 left) 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout:398: rollback oid 37 current snap is 37 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout:rollback oid 37 to 21 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout:399: read oid 25 snap -1 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout:399: expect (ObjNum 127 snap 27 seq_num 127) 2024-09-16T02:15:39.865 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 2 2024-09-16T02:15:39.873 INFO:tasks.rados.rados.0.smithi195.stdout:398: finishing rollback tid 0 to smithi19530025-37 2024-09-16T02:15:39.874 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:39.873+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:39.874 INFO:tasks.rados.rados.0.smithi195.stdout:398: finishing rollback tid 1 to smithi19530025-37 2024-09-16T02:15:39.877 INFO:tasks.rados.rados.0.smithi195.stdout:398: finishing rollback tid 2 to smithi19530025-37 2024-09-16T02:15:39.877 INFO:tasks.rados.rados.0.smithi195.stdout:update_object_version oid 37 v 138 (ObjNum 76 snap 8 seq_num 76) dirty exists 2024-09-16T02:15:39.877 INFO:tasks.rados.rados.0.smithi195.stdout:398: done (1 left) 2024-09-16T02:15:39.877 INFO:tasks.rados.rados.0.smithi195.stdout: waiting on 1 2024-09-16T02:15:39.882 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:39.883 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:39.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:39.911 INFO:tasks.rados.rados.0.smithi195.stdout:399: expect (ObjNum 127 snap 27 seq_num 127) 2024-09-16T02:15:39.912 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:39.975 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:39.970+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:39.989 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:40.014 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:40.021 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.021 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:40.067 INFO:tasks.rados.rados.0.smithi195.stdout:seeking to 0 2024-09-16T02:15:40.083 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:40.080+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:40.118 INFO:tasks.rados.rados.0.smithi195.stdout:399: done (0 left) 2024-09-16T02:15:40.121 INFO:tasks.rados.rados.0.smithi195.stderr:0 errors. 2024-09-16T02:15:40.121 INFO:tasks.rados.rados.0.smithi195.stderr: 2024-09-16T02:15:40.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:15:40.160 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.160 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:40.178 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:40.178+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:40.276 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:40.274+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:40.341 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.341 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.355 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:40.377 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:40.374+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:40.463 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:15:40.463 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":111,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:15:39.980508+0000","last_up_change":"2024-09-16T02:15:28.919687+0000","last_in_change":"2024-09-16T02:15:12.022965+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.9375,"average_primary_affinity":1,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3.75,"score_stable":3.75,"optimal_score":0.60000002384185791,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.9375,"average_primary_affinity":1,"average_primary_affinity_weighted":0.93999999761581421}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-16T02:14:20.234400+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":3,"size":7,"min_size":5,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":39,"snap_epoch":110,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"shecprofile","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":16384,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":81,"up_from":100,"up_thru":32,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6809","nonce":3610577698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6811","nonce":3610577698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6815","nonce":3610577698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6813","nonce":3610577698}]},"public_addr":"172.21.15.177:6809/3610577698","cluster_addr":"172.21.15.177:6811/3610577698","heartbeat_back_addr":"172.21.15.177:6815/3610577698","heartbeat_front_addr":"172.21.15.177:6813/3610577698","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:06.081345+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:20.891965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:09.212164+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:16.290057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:10.699837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:09.304914+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:04.717114+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:15.023066+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:18.765693+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:25.090766+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-16T02:15:28.101178+0000","dead_epoch":82},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:17.826946+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:19.298920+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.709317+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.987870+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:19.677375+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.9","osds":[7,13,1,0,8,6,12]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":29,"length":1},{"begin":31,"length":1},{"begin":34,"length":3},{"begin":38,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:15:40.472 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.472 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.478 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:40.472+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:40.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:15:40.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:40.579 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:40.576+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:40.644 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.644 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:40.679 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:40.678+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:40.780 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:40.776+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:40.821 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.821 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:40.835 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:40.849 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:15:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:15:40.869 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-16T02:15:40.021277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112411187,"num_objects":88,"num_object_clones":37,"num_object_copies":596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":5,"num_read":513,"num_read_kb":140598,"num_write":1037,"num_write_kb":155490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1371,"ondisk_log_size":1371,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":661204,"kb_used_data":205808,"kb_used_omap":23,"kb_used_meta":455016,"kb_avail":1414916396,"statfs":{"total":1449551462400,"available":1448874389504,"internally_reserved":0,"allocated":210747392,"data_stored":260621431,"data_compressed":3808159,"data_compressed_allocated":5234688,"data_compressed_original":11329536,"omap_allocated":23922,"internal_metadata":465937038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8929,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001219"},"pg_stats":[{"pgid":"3.f","version":"106'47","reported_seq":397,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.998790+0000","last_change":"2024-09-16T02:15:35.000286+0000","last_active":"2024-09-16T02:15:34.998790+0000","last_peered":"2024-09-16T02:15:34.998790+0000","last_clean":"2024-09-16T02:15:34.998790+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:15:34.998790+0000","last_undegraded":"2024-09-16T02:15:34.998790+0000","last_fullsized":"2024-09-16T02:15:34.998790+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:38.125773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027392118999999999,"stat_sum":{"num_bytes":4751360,"num_objects":4,"num_object_clones":1,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":2664,"num_write":18,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.e","version":"106'79","reported_seq":549,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.033992+0000","last_change":"2024-09-16T02:15:38.989889+0000","last_active":"2024-09-16T02:15:39.033992+0000","last_peered":"2024-09-16T02:15:39.033992+0000","last_clean":"2024-09-16T02:15:39.033992+0000","last_became_active":"2024-09-16T02:15:13.070271+0000","last_became_peered":"2024-09-16T02:15:13.070271+0000","last_unstale":"2024-09-16T02:15:39.033992+0000","last_undegraded":"2024-09-16T02:15:39.033992+0000","last_fullsized":"2024-09-16T02:15:39.033992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'66","last_scrub_stamp":"2024-09-16T02:15:27.680026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.680026+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.051994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00059237999999999999,"stat_sum":{"num_bytes":11616256,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":18552,"num_write":73,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,14,11,0,15],"acting":[8,9,1,14,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.d","version":"106'110","reported_seq":453,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.007990+0000","last_change":"2024-09-16T02:15:35.009149+0000","last_active":"2024-09-16T02:15:35.007990+0000","last_peered":"2024-09-16T02:15:35.007990+0000","last_clean":"2024-09-16T02:15:35.007990+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:15:35.007990+0000","last_undegraded":"2024-09-16T02:15:35.007990+0000","last_fullsized":"2024-09-16T02:15:35.007990+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:48.883671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041628440000000003,"stat_sum":{"num_bytes":5488640,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":15024,"num_write":58,"num_write_kb":12320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.c","version":"99'31","reported_seq":411,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.986081+0000","last_change":"2024-09-16T02:15:38.986870+0000","last_active":"2024-09-16T02:15:38.986081+0000","last_peered":"2024-09-16T02:15:38.986081+0000","last_clean":"2024-09-16T02:15:38.986081+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:15:38.986081+0000","last_undegraded":"2024-09-16T02:15:38.986081+0000","last_fullsized":"2024-09-16T02:15:38.986081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'29","last_scrub_stamp":"2024-09-16T02:15:27.850068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.850068+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:50.020929+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0014552569999999999,"stat_sum":{"num_bytes":3162112,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7055,"num_write":13,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.b","version":"106'71","reported_seq":452,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.996041+0000","last_change":"2024-09-16T02:15:35.001752+0000","last_active":"2024-09-16T02:15:34.996041+0000","last_peered":"2024-09-16T02:15:34.996041+0000","last_clean":"2024-09-16T02:15:34.996041+0000","last_became_active":"2024-09-16T02:15:13.074136+0000","last_became_peered":"2024-09-16T02:15:13.074136+0000","last_unstale":"2024-09-16T02:15:34.996041+0000","last_undegraded":"2024-09-16T02:15:34.996041+0000","last_fullsized":"2024-09-16T02:15:34.996041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:50.249432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026872966000000002,"stat_sum":{"num_bytes":10207232,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":12153,"num_write":53,"num_write_kb":10416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8077312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,2,13],"acting":[7,15,1,9,8,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.968243+0000","last_change":"2024-09-16T02:15:34.969093+0000","last_active":"2024-09-16T02:15:34.968243+0000","last_peered":"2024-09-16T02:15:34.968243+0000","last_clean":"2024-09-16T02:15:34.968243+0000","last_became_active":"2024-09-16T02:15:13.070217+0000","last_became_peered":"2024-09-16T02:15:13.070217+0000","last_unstale":"2024-09-16T02:15:34.968243+0000","last_undegraded":"2024-09-16T02:15:34.968243+0000","last_fullsized":"2024-09-16T02:15:34.968243+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001402737,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,14,1,9,8,15,2],"acting":[6,14,1,9,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.9","version":"106'28","reported_seq":287,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.990896+0000","last_change":"2024-09-16T02:15:34.993249+0000","last_active":"2024-09-16T02:15:34.990896+0000","last_peered":"2024-09-16T02:15:34.990896+0000","last_clean":"2024-09-16T02:15:34.990896+0000","last_became_active":"2024-09-16T02:14:43.948134+0000","last_became_peered":"2024-09-16T02:14:43.948134+0000","last_unstale":"2024-09-16T02:15:34.990896+0000","last_undegraded":"2024-09-16T02:15:34.990896+0000","last_fullsized":"2024-09-16T02:15:34.990896+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'26","last_scrub_stamp":"2024-09-16T02:15:30.066659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:30.066659+0000","objects_scrubbed":2,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:37.191474+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.018542082000000001,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13,1,0,8,6,12],"acting":[7,13,1,0,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.8","version":"106'121","reported_seq":480,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T02:15:34.991958+0000","last_change":"2024-09-16T02:15:34.991958+0000","last_active":"2024-09-16T02:15:34.991958+0000","last_peered":"2024-09-16T02:15:34.991958+0000","last_clean":"2024-09-16T02:15:34.991958+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:15:34.991958+0000","last_undegraded":"2024-09-16T02:15:34.991958+0000","last_fullsized":"2024-09-16T02:15:34.991958+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.428155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015878369,"stat_sum":{"num_bytes":9732096,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":18184,"num_write":67,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"1.0","version":"26'112","reported_seq":307,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.979268+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:15:34.979268+0000","last_peered":"2024-09-16T02:15:34.979268+0000","last_clean":"2024-09-16T02:15:34.979268+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:15:34.979268+0000","last_undegraded":"2024-09-16T02:15:34.979268+0000","last_fullsized":"2024-09-16T02:15:34.979268+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":177,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.992783+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:15:34.992783+0000","last_peered":"2024-09-16T02:15:34.992783+0000","last_clean":"2024-09-16T02:15:34.992783+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:15:34.992783+0000","last_undegraded":"2024-09-16T02:15:34.992783+0000","last_fullsized":"2024-09-16T02:15:34.992783+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"106'157","reported_seq":565,"reported_epoch":106,"state":"active+clean+snaptrim","last_fresh":"2024-09-16T02:15:35.000855+0000","last_change":"2024-09-16T02:15:34.980685+0000","last_active":"2024-09-16T02:15:35.000855+0000","last_peered":"2024-09-16T02:15:35.000855+0000","last_clean":"2024-09-16T02:15:35.000855+0000","last_became_active":"2024-09-16T02:15:13.072281+0000","last_became_peered":"2024-09-16T02:15:13.072281+0000","last_unstale":"2024-09-16T02:15:35.000855+0000","last_undegraded":"2024-09-16T02:15:35.000855+0000","last_fullsized":"2024-09-16T02:15:35.000855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:44.382777+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017367517999999998,"stat_sum":{"num_bytes":17940480,"num_objects":13,"num_object_clones":6,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":68,"num_read_kb":17164,"num_write":128,"num_write_kb":21536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,4,14,1,0],"acting":[13,12,6,4,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"2"},{"start":"22","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":174,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.971989+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:15:34.971989+0000","last_peered":"2024-09-16T02:15:34.971989+0000","last_clean":"2024-09-16T02:15:34.971989+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:15:34.971989+0000","last_undegraded":"2024-09-16T02:15:34.971989+0000","last_fullsized":"2024-09-16T02:15:34.971989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:25.996964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"106'228","reported_seq":623,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.059109+0000","last_change":"2024-09-16T02:15:35.060305+0000","last_active":"2024-09-16T02:15:35.059109+0000","last_peered":"2024-09-16T02:15:35.059109+0000","last_clean":"2024-09-16T02:15:35.059109+0000","last_became_active":"2024-09-16T02:14:58.940506+0000","last_became_peered":"2024-09-16T02:14:58.940506+0000","last_unstale":"2024-09-16T02:15:35.059109+0000","last_undegraded":"2024-09-16T02:15:35.059109+0000","last_fullsized":"2024-09-16T02:15:35.059109+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'209","last_scrub_stamp":"2024-09-16T02:15:29.441137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:29.441137+0000","objects_scrubbed":13,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:29.660064+0000","scrub_duration":23,"objects_trimmed":1,"snaptrim_duration":0.073047173000000007,"stat_sum":{"num_bytes":18038784,"num_objects":12,"num_object_clones":6,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":53,"num_read_kb":17941,"num_write":186,"num_write_kb":24992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":171,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.979406+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:15:34.979406+0000","last_peered":"2024-09-16T02:15:34.979406+0000","last_clean":"2024-09-16T02:15:34.979406+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:15:34.979406+0000","last_undegraded":"2024-09-16T02:15:34.979406+0000","last_fullsized":"2024-09-16T02:15:34.979406+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:20.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"48'8","reported_seq":351,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.967490+0000","last_change":"2024-09-16T02:15:34.968351+0000","last_active":"2024-09-16T02:15:34.967490+0000","last_peered":"2024-09-16T02:15:34.967490+0000","last_clean":"2024-09-16T02:15:34.967490+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:15:34.967490+0000","last_undegraded":"2024-09-16T02:15:34.967490+0000","last_fullsized":"2024-09-16T02:15:34.967490+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013024899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":184,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.972341+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:15:34.972341+0000","last_peered":"2024-09-16T02:15:34.972341+0000","last_clean":"2024-09-16T02:15:34.972341+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:15:34.972341+0000","last_undegraded":"2024-09-16T02:15:34.972341+0000","last_fullsized":"2024-09-16T02:15:34.972341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"106'29","reported_seq":438,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.999643+0000","last_change":"2024-09-16T02:15:35.001733+0000","last_active":"2024-09-16T02:15:34.999643+0000","last_peered":"2024-09-16T02:15:34.999643+0000","last_clean":"2024-09-16T02:15:34.999643+0000","last_became_active":"2024-09-16T02:15:13.074572+0000","last_became_peered":"2024-09-16T02:15:13.074572+0000","last_unstale":"2024-09-16T02:15:34.999643+0000","last_undegraded":"2024-09-16T02:15:34.999643+0000","last_fullsized":"2024-09-16T02:15:34.999643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.021639180000000001,"stat_sum":{"num_bytes":1687552,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":5328,"num_write":29,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,12],"acting":[13,2,11,14,4,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"3.5","version":"106'66","reported_seq":155,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.985612+0000","last_change":"2024-09-16T02:15:34.987430+0000","last_active":"2024-09-16T02:15:34.985612+0000","last_peered":"2024-09-16T02:15:34.985612+0000","last_clean":"2024-09-16T02:15:34.985612+0000","last_became_active":"2024-09-16T02:15:13.073157+0000","last_became_peered":"2024-09-16T02:15:13.073157+0000","last_unstale":"2024-09-16T02:15:34.985612+0000","last_undegraded":"2024-09-16T02:15:34.985612+0000","last_fullsized":"2024-09-16T02:15:34.985612+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020256013999999999,"stat_sum":{"num_bytes":8912896,"num_objects":5,"num_object_clones":1,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":21195,"num_write":34,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5144576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,15,14,0,7,8],"acting":[12,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":88,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.987473+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:15:34.987473+0000","last_peered":"2024-09-16T02:15:34.987473+0000","last_clean":"2024-09-16T02:15:34.987473+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:15:34.987473+0000","last_undegraded":"2024-09-16T02:15:34.987473+0000","last_fullsized":"2024-09-16T02:15:34.987473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"106'98","reported_seq":478,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.018843+0000","last_change":"2024-09-16T02:15:35.019924+0000","last_active":"2024-09-16T02:15:35.018843+0000","last_peered":"2024-09-16T02:15:35.018843+0000","last_clean":"2024-09-16T02:15:35.018843+0000","last_became_active":"2024-09-16T02:14:55.933032+0000","last_became_peered":"2024-09-16T02:14:55.933032+0000","last_unstale":"2024-09-16T02:15:35.018843+0000","last_undegraded":"2024-09-16T02:15:35.018843+0000","last_fullsized":"2024-09-16T02:15:35.018843+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:41.156017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.043122655000000003,"stat_sum":{"num_bytes":4014080,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":1359,"num_write":93,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4112384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":182,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.989102+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:15:38.989102+0000","last_peered":"2024-09-16T02:15:38.989102+0000","last_clean":"2024-09-16T02:15:38.989102+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:15:38.989102+0000","last_undegraded":"2024-09-16T02:15:38.989102+0000","last_fullsized":"2024-09-16T02:15:38.989102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:21.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"106'134","reported_seq":470,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:35.004474+0000","last_change":"2024-09-16T02:15:35.004474+0000","last_active":"2024-09-16T02:15:35.004474+0000","last_peered":"2024-09-16T02:15:35.004474+0000","last_clean":"2024-09-16T02:15:35.004474+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:15:35.004474+0000","last_undegraded":"2024-09-16T02:15:35.004474+0000","last_fullsized":"2024-09-16T02:15:35.004474+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.026077304999999999,"stat_sum":{"num_bytes":9994240,"num_objects":9,"num_object_clones":4,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":2074,"num_write":82,"num_write_kb":13776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d","length":"1"},{"start":"1b","length":"3"},{"start":"22","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":191,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.994121+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:15:38.994121+0000","last_peered":"2024-09-16T02:15:38.994121+0000","last_clean":"2024-09-16T02:15:38.994121+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:15:38.994121+0000","last_undegraded":"2024-09-16T02:15:38.994121+0000","last_fullsized":"2024-09-16T02:15:38.994121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"110'49","reported_seq":434,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.004424+0000","last_change":"2024-09-16T02:15:39.006345+0000","last_active":"2024-09-16T02:15:39.004424+0000","last_peered":"2024-09-16T02:15:39.004424+0000","last_clean":"2024-09-16T02:15:39.004424+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:15:39.004424+0000","last_undegraded":"2024-09-16T02:15:39.004424+0000","last_fullsized":"2024-09-16T02:15:39.004424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'45","last_scrub_stamp":"2024-09-16T02:15:36.983585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:36.983585+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:53.480085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.012849442000000001,"stat_sum":{"num_bytes":2506752,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":1828,"num_write":36,"num_write_kb":2736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":182,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-09-16T02:15:34.967835+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:15:34.967835+0000","last_peered":"2024-09-16T02:15:34.967835+0000","last_clean":"2024-09-16T02:15:34.967835+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:15:34.967835+0000","last_undegraded":"2024-09-16T02:15:34.967835+0000","last_fullsized":"2024-09-16T02:15:34.967835+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":111820800,"num_objects":84,"num_object_clones":37,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":5,"num_read":427,"num_read_kb":140525,"num_write":890,"num_write_kb":152816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202108928,"data_stored":255725568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},"log_size":1256,"ondisk_log_size":1256,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47560,"kb_used_data":12208,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324280,"statfs":{"total":96636764160,"available":96588062720,"internally_reserved":0,"allocated":12500992,"data_stored":17089001,"data_compressed":279437,"data_compressed_allocated":380928,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280533,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44528,"kb_used_data":13332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327312,"statfs":{"total":96636764160,"available":96591167488,"internally_reserved":0,"allocated":13651968,"data_stored":17502153,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280534,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41012,"kb_used_data":9824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330828,"statfs":{"total":96636764160,"available":96594767872,"internally_reserved":0,"allocated":10059776,"data_stored":12771273,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313239,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49920,"kb_used_data":18732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321920,"statfs":{"total":96636764160,"available":96585646080,"internally_reserved":0,"allocated":19181568,"data_stored":23839561,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46984,"kb_used_data":15788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324856,"statfs":{"total":96636764160,"available":96588652544,"internally_reserved":0,"allocated":16166912,"data_stored":20910025,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48472,"kb_used_data":17284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":17698816,"data_stored":22803273,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41772,"kb_used_data":10580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330068,"statfs":{"total":96636764160,"available":96593989632,"internally_reserved":0,"allocated":10833920,"data_stored":13400230,"data_compressed":249942,"data_compressed_allocated":344064,"data_compressed_original":688128,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280533,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36968,"kb_used_data":9932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334872,"statfs":{"total":96636764160,"available":96598908928,"internally_reserved":0,"allocated":10170368,"data_stored":11517897,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280532,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46156,"kb_used_data":14960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325684,"statfs":{"total":96636764160,"available":96589500416,"internally_reserved":0,"allocated":15319040,"data_stored":18751452,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39760,"kb_used_data":12736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332080,"statfs":{"total":96636764160,"available":96596049920,"internally_reserved":0,"allocated":13041664,"data_stored":16335689,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50876,"kb_used_data":19684,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320964,"statfs":{"total":96636764160,"available":96584667136,"internally_reserved":0,"allocated":20156416,"data_stored":24588777,"data_compressed":279437,"data_compressed_allocated":380928,"data_compressed_original":1191936,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43788,"kb_used_data":12592,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328052,"statfs":{"total":96636764160,"available":96591925248,"internally_reserved":0,"allocated":12894208,"data_stored":14581705,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280533,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37324,"kb_used_data":10292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334516,"statfs":{"total":96636764160,"available":96598544384,"internally_reserved":0,"allocated":10539008,"data_stored":12214217,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280533,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45968,"kb_used_data":14780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325872,"statfs":{"total":96636764160,"available":96589692928,"internally_reserved":0,"allocated":15134720,"data_stored":18501577,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280533,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40116,"kb_used_data":13084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94331724,"statfs":{"total":96636764160,"available":96595685376,"internally_reserved":0,"allocated":13398016,"data_stored":15814601,"data_compressed":246906,"data_compressed_allocated":339968,"data_compressed_original":679936,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":0,"seq":429496729603,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18874368,"data_stored":22790144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":14946304,"data_stored":19677184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9596928,"data_stored":12472320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":14692352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9318400,"data_stored":11005952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9863168,"data_stored":11628544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13914112,"data_stored":17207296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":15007744,"data_stored":18923520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11816960,"data_stored":16326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":12177408,"data_stored":14811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7139328,"data_stored":8015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8839168,"data_stored":11792384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12431360,"data_stored":16293888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11218944,"data_stored":15327232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":17948672,"data_stored":22917120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":16429056,"data_stored":21843968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:15:40.871 INFO:tasks.ceph.ceph_manager.ceph:pg 3.8 in trimming, state: active+clean+snaptrim 2024-09-16T02:15:40.871 INFO:tasks.ceph.ceph_manager.ceph:pg 3.2 in trimming, state: active+clean+snaptrim 2024-09-16T02:15:40.871 INFO:tasks.ceph.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-09-16T02:15:40.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:40.914 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:40.914+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:40.983 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:40.978+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:40.996 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:40.996 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:41.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:41.076 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:41.083 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:41.078+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:41.100 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:41.100 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:41.130 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:41.130 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:41.132 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:41.132 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:41.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:41.161 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:41.161 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:41.184 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:41.182+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:41.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:41.222 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:41.256 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:41.256 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:41.272 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:41.272 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:41.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:41.286 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:41.282+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:41.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:41.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:41.319 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:41.320 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:41.352 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:41.386 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:41.382+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:41.387 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:41.387 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:41.403 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:41.403 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:41.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:41.424 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:41.457 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:41.486 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:41.486+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:41.491 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:41.492 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:41.528 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:41.526+0000 7f02ded9d640 0 -- 172.21.15.3:0/2697531914 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f02e02275a0 msgr2=0x7f02e0229990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:41.544 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 95.470070835000001, 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:15:41.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:15:41.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:41.552 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 94.632090636000001, 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:15:41.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:15:41.555 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.556 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:15:41.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.263519458000005, 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:15:41.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.560 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:15:41.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.562 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.563 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.260041064999996, 2024-09-16T02:15:41.564 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.566 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.567 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:15:41.568 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.569 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.570 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.258398056000004, 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.571 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.572 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.573 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:15:41.574 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:15:41.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.576 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.577 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.258301837000005, 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.578 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:15:41.579 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.580 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:15:41.581 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.582 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.583 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 77.225463415999997, 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:15:41.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.585 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.587 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:15:41.588 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 67.173727669000002, 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.589 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.590 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.591 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.592 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 67.169439988999997, 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.593 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.594 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:15:41.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:15:41.596 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.474885426, 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27886435399999998, 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.597 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.598 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066449+0000", 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.435e-06 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066471+0000", 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1526999999999999e-05 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343774+0000", 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27730348599999999 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343789+0000", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4946000000000001e-05 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343981+0000", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019161199999999999 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.344000+0000", 2024-09-16T02:15:41.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545e-05 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345225+0000", 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012244479999999999 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345312+0000", 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7355000000000006e-05 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.601 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 84/83 ECSubWriteReply(tid=304, last_complete=84'106, committed=1, applied=1))", 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.377776892, 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67482556900000001, 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:41.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163558+0000", 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163565+0000", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5129999999999999e-06 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837152+0000", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67358686899999998 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837166+0000", 2024-09-16T02:15:41.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3685e-05 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837280+0000", 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114374 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.838382+0000", 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011018079999999999 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.447901614999999, 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59632199500000005, 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:41.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093434+0000", 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.39e-06 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093480+0000", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5844999999999999e-05 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688435+0000", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59495508799999997 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688450+0000", 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5345999999999998e-05 2024-09-16T02:15:41.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688617+0000", 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166608 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688648+0000", 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1282000000000003e-05 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689677+0000", 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010285520000000001 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689753+0000", 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6884000000000005e-05 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.610 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:868 3.3s0 3:c628ba0c:::smithi19530025-27:head [copy-from ver 49] snapc 23=[23,22,21,20,1f,1e,1d,18,d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.618583441, 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042527884000000002, 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 868 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922753+0000", 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.937e-06 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922766+0000", 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3849000000000001e-05 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922800+0000", 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3325999999999999e-05 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922818+0000", 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7782e-05 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922967+0000", 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000149605 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922980+0000", 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3159999999999999e-05 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931241+0000", 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082608520000000008 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931506+0000", 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026437699999999999 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.957452+0000", 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025946005000000001 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.962422+0000", 2024-09-16T02:15:41.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049703719999999998 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.965127+0000", 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027051079999999999 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.965278+0000", 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015051100000000001 2024-09-16T02:15:41.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:869 3.3s0 3:c628ba0c:::smithi19530025-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 14.618468125, 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038142744999999999, 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 869 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:41.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922867+0000", 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9039999999999999e-06 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922875+0000", 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9659999999999996e-06 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922896+0000", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1433e-05 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922913+0000", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6415000000000001e-05 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.923098+0000", 2024-09-16T02:15:41.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000185014 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.923107+0000", 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6560000000000004e-06 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931923+0000", 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008816127 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.932051+0000", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012747799999999999 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.960518+0000", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02846655 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.960769+0000", 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025148100000000001 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.961008+0000", 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023872100000000001 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDScrubReserve(3.2s0,GRANT,e:108,from:14(4),reservation_nonce:0)", 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4.2987414360000002, 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.698473758, 2024-09-16T02:15:41.624 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:41.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242593+0000", 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242604+0000", 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0285999999999999e-05 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940227+0000", 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.69762373099999997 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940241+0000", 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3616e-05 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940507+0000", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026620000000000002 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940560+0000", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2621000000000003e-05 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.941065+0000", 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050548699999999999 2024-09-16T02:15:41.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:968 3.2s0 3:45ca573f:::smithi19530025-10:head [stat,delete] snapc 27=[26,25,21,20,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.6137166450000002, 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047565187000000002, 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 968 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:41.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927619+0000", 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8229999999999999e-06 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927630+0000", 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927663+0000", 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2669000000000002e-05 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927680+0000", 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6796e-05 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927844+0000", 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016416600000000001 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927861+0000", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6426000000000002e-05 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.931526+0000", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003664984 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.931809+0000", 2024-09-16T02:15:41.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028384500000000002 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.934082+0000", 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022728620000000001 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.938022+0000", 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039393320000000003 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.974917+0000", 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036895178000000001 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.975182+0000", 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026487599999999999 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 109/83 ECSubWriteReply(tid=386, last_complete=109'165, committed=1, applied=1))", 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.5560683759999998, 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057427276999999999, 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4300000000000002e-07 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985270+0000", 2024-09-16T02:15:41.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0570000000000003e-06 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040852+0000", 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055582661999999998 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040867+0000", 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4450000000000001e-05 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.042636+0000", 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001769561 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.042692+0000", 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5804000000000003e-05 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 109/83 ECSubWriteReply(tid=389, last_complete=109'169, committed=1, applied=1))", 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.4930965239999998, 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.116722672, 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048238+0000", 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.863e-06 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048251+0000", 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2556e-05 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162830+0000", 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.114579261 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162849+0000", 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8564e-05 2024-09-16T02:15:41.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.164639+0000", 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017899229999999999 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.164959+0000", 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032050500000000001 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:984 3.2s0 3:476b4430:::smithi19530025-30:head [setxattr _header (57) in=7b] snapc 27=[26,25,21,20,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.3835645469999998, 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032437222000000002, 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.645 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 984 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157771+0000", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1799999999999999e-06 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157781+0000", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0013000000000001e-05 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157799+0000", 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7853e-05 2024-09-16T02:15:41.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157809+0000", 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0849000000000001e-05 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162247+0000", 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044376770000000001 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162526+0000", 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000278869 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.163578+0000", 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010525129999999999 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165178+0000", 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001599105 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.189999+0000", 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024821288 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.190206+0000", 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206875 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.650 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:985 3.2s0 3:476b4430:::smithi19530025-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 3.3834954650000002, 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035613218000000002, 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 985 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.651 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:41.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157839+0000", 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.279e-06 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157845+0000", 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0229999999999998e-06 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157864+0000", 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9317000000000002e-05 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157877+0000", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3149000000000001e-05 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165341+0000", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074638450000000002 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165514+0000", 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017282800000000001 2024-09-16T02:15:41.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.191138+0000", 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025624257000000001 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.191391+0000", 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025285299999999999 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:41.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.193451+0000", 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020596669999999998 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:41.656 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:41.658 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:41.658 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:41.652+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:41.759 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:41.752+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:41.769 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:41.769 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:41.790 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:41.791 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:41.791 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:41.791 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:41.805 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:41.860 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:41.858+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:41.924 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:41.924 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:41.924 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:41.924 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:41.924 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:41.938 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:41.961 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:41.958+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:42.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-16T02:15:42.062 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:42.058+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:42.063 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:42.063 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:42.063 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 93.871178932999996, 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:42.064 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.065 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.066 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:15:42.067 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:15:42.067 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.067 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.067 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.067 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:15:42.068 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:15:42.068 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.068 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.068 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.068 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:15:42.069 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:15:42.070 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.070 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.070 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:15:42.070 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:15:42.070 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:15:42.071 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.071 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.071 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.071 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:15:42.071 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.072 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 68.718587501000002, 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095323660000000004, 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.073 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343124+0000", 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.06e-06 2024-09-16T02:15:42.074 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343143+0000", 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8842e-05 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343322+0000", 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179414 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.075 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343345+0000", 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2629e-05 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343569+0000", 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022398900000000001 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343596+0000", 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6489e-05 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.076 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352536+0000", 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089404250000000001 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352656+0000", 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119518 2024-09-16T02:15:42.077 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/32 ECSubWrite(tid=19, reqid=osd.6.0:19, at_version=44'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 68.718498253999996, 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083893449999999994, 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.078 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.079 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.080 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.080 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343214+0000", 2024-09-16T02:15:42.080 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-16T02:15:42.080 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.080 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.081 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.081 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343264+0000", 2024-09-16T02:15:42.081 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0175e-05 2024-09-16T02:15:42.081 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.081 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343378+0000", 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114159 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.082 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343396+0000", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7926000000000001e-05 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343513+0000", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117056 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343528+0000", 2024-09-16T02:15:42.083 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4324999999999999e-05 2024-09-16T02:15:42.084 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.084 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.084 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.084 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351501+0000", 2024-09-16T02:15:42.084 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079735710000000005 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351602+0000", 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010059 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:15:42.085 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 68.717163342999996, 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082631800000000002, 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.086 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344549+0000", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344608+0000", 2024-09-16T02:15:42.087 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8953999999999998e-05 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344655+0000", 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7268999999999998e-05 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344677+0000", 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1382e-05 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.088 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344840+0000", 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016295099999999999 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344865+0000", 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5035e-05 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352714+0000", 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078496989999999999 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.089 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352810+0000", 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6146999999999999e-05 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.714347997000004, 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009462359, 2024-09-16T02:15:42.090 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.091 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347364+0000", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.623e-06 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347386+0000", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1667000000000001e-05 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347450+0000", 2024-09-16T02:15:42.092 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4116999999999996e-05 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347464+0000", 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4347e-05 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348099+0000", 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063462999999999996 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.093 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348122+0000", 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2866000000000001e-05 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356742+0000", 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086204220000000008 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.094 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356825+0000", 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2687000000000003e-05 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 45/32 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=45'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 67.710438119000003, 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010034855000000001, 2024-09-16T02:15:42.095 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.096 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351322+0000", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351430+0000", 2024-09-16T02:15:42.097 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108572 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351458+0000", 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8229e-05 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351962+0000", 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050377899999999999 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.098 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351989+0000", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6322e-05 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361119+0000", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091308310000000007 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361307+0000", 2024-09-16T02:15:42.099 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188054 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:394 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.154390159000002, 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094620629999999997, 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.100 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 394 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.101 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907323+0000", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5399999999999998e-06 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907336+0000", 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2605999999999999e-05 2024-09-16T02:15:42.102 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907364+0000", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8538999999999999e-05 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907387+0000", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979000000000001e-05 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907499+0000", 2024-09-16T02:15:42.103 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011208 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907666+0000", 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016685800000000001 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.909169+0000", 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015026659999999999 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.104 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914888+0000", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057194580000000002 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916546+0000", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001657911 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916783+0000", 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236426 2024-09-16T02:15:42.105 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:395 3.fs0 3:f97ec54c:::smithi19530025-26:head [rollback 8] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.154285848999997, 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011190179, 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.106 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 395 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.107 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907426+0000", 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907433+0000", 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.108 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907453+0000", 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907485+0000", 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2066000000000001e-05 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910164+0000", 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026789510000000002 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.109 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910300+0000", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013539300000000001 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.911564+0000", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001264577 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.917774+0000", 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062099729999999997 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.110 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918506+0000", 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073229000000000002 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918615+0000", 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108527 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.111 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:396 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 61.154199794999997, 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011749021, 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 396 2024-09-16T02:15:42.112 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.113 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907512+0000", 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1060000000000001e-06 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907519+0000", 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.917e-06 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.114 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907537+0000", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907552+0000", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912502+0000", 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049500600000000001 2024-09-16T02:15:42.115 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912741+0000", 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023893499999999999 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914057+0000", 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013153119999999999 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.116 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918884+0000", 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048277240000000003 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919157+0000", 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306100000000001 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919260+0000", 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102594 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.117 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.069865069999999, 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091235069999999995, 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.118 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991847+0000", 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0699999999999999e-06 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.119 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991900+0000", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3204000000000001e-05 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992025+0000", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012476300000000001 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992053+0000", 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8085e-05 2024-09-16T02:15:42.120 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992229+0000", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017621299999999999 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992255+0000", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6319e-05 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000887+0000", 2024-09-16T02:15:42.121 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086322410000000006 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000969+0000", 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1612000000000004e-05 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(osd.7.22:3 3.fs0 3:f2f6c7ec:::smithi19530025-24:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.020878070000002, 2024-09-16T02:15:42.122 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090619859999999993, 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.123 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040836+0000", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4659999999999999e-06 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040855+0000", 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8689e-05 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.124 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040906+0000", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1381000000000002e-05 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040930+0000", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3739000000000001e-05 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041122+0000", 2024-09-16T02:15:42.125 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191669 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041284+0000", 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000162917 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.049895+0000", 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086101249999999997 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.126 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 27.968220082999999, 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50299762999999997, 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.127 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093493+0000", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2400000000000002e-06 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093538+0000", 2024-09-16T02:15:42.128 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4971e-05 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595224+0000", 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50168629200000003 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595240+0000", 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.56e-05 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.129 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595366+0000", 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012640000000000001 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595382+0000", 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5936000000000001e-05 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596420+0000", 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010379499999999999 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.130 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596488+0000", 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8240999999999999e-05 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 103/83 ECSubWrite(tid=356, reqid=osd.13.0:356, at_version=103'25, trim_to=0'0, roll_forward_to=98'23))", 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.117469378999999, 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079670350000000008, 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.131 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.132 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944242+0000", 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944284+0000", 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1752000000000003e-05 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.133 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.946707+0000", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024230660000000002 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.946731+0000", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4774999999999999e-05 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.947311+0000", 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057995599999999996 2024-09-16T02:15:42.134 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.947338+0000", 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6803000000000001e-05 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.952113+0000", 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047752300000000001 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.135 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.952208+0000", 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4877000000000007e-05 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:957 3.fs0 3:f560586e:::smithi19530025-12:head [create,setxattr clNOgfqtsg\\aUXcbnbsgicte[dlcgscTctZg\\UYb (1261) in=40b,setxattr ^kePZO^jc[`nRrZQr^g`hWUbUbmVa_[^`ndQ_ebi (207) in=40b,setxattr pd\\tReSgkdQqSk_oQZgRskailqSfXkleakRgOpgs (735) in=40b,setxattr `\\Uhenhpc\\oVlscgQb^Wpj]b[jPkrs]S`tY^nfNX (1506) in=40b,setxattr mYhN^Rig]VasbVqh]mfR[[pqUcOYh]_aWp\\m_tls (1891) in=40b,setxattr hie`]lstsZtOilXYaRecaiXtt]gcmm^a[tWjt[YT (311) in=40b,setxattr hkRZhYOPaigVYZmnZe^p`NTbkSNSiVgfgVijksca (455) in=40b,setxattr `sdWbhfe[ng_eOr]YcXa]ahtbPgochac``jdt`hh (1262) in=40b,setxattr aPlreWi_Xm]iktYWh`g_aSaRPrXe]adtOdO]]bsq (347) in=40b,setxattr p_qb[TbOoSb^Z[jcnVcp[Vb_XQ^gddoWiVa^NPYR (385) in=40b,setxattr VgQd`_b]^jekaYilkQoqdami]]mhVp\\fbWfmkbVU (1295) in=40b,setxattr tWfY]UYjtVla^t`]`Z[s\\sp`]jp^jcoVP[kaYTjb (1893) in=40b,setxattr ZeWXgW`YOnsgbbrr`eRYbpbOQeTm_g^hkeqUdl_X (982) in=40b,setxattr aoh__jnjZmcV_Zd[lTjob_PfRoWV (1822) in=28b,setxattr Og\\PgsWdl_jTZ]kbbRXdO`bhQsdfmNUjgURgh\\`^ (1997) in=40b,setxattr fWcmbPRf_fT[s`o`jq^\\U_m_UlTQhjpPfe]Oq^Zs (1645) in=40b,setxattr jQT^\\Zie\\orcqPOr]sg\\\\PWNdY`OYs^QYktWQsTd (1789) in=40b,setxattr N_TeWQ`\\cU]`mQPYod^P]RXe[YaVQncV^sk`_bVn (1785) in=40b,setxattr jQhOSafV (1822) in=8b,setxattr ]daaodicjtV`q_`__etNiom_n`abiaa`sbrXmcaT (947) in=40b] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1370752819999996, 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026264025999999999, 2024-09-16T02:15:42.136 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 957 2024-09-16T02:15:42.137 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.138 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924639+0000", 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.32e-06 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924738+0000", 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8863999999999994e-05 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924768+0000", 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0669000000000001e-05 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.139 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924786+0000", 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7977000000000001e-05 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924949+0000", 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000162325 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:15:42.140 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924971+0000", 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2079e-05 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.934469+0000", 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094982409999999993 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.934801+0000", 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033254299999999998 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.141 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.937819+0000", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003017568 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949768+0000", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011948780000000001 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949987+0000", 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021956999999999999 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.142 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.950899+0000", 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091208999999999995 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=128, reqid=osd.6.0:128, at_version=107'114, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:42.143 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1347398049999997, 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017360448000000001, 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.144 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926973+0000", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.72e-06 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927024+0000", 2024-09-16T02:15:42.145 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1595000000000003e-05 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927136+0000", 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011198 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927160+0000", 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3462999999999999e-05 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.146 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927318+0000", 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015849200000000001 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927336+0000", 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7538999999999999e-05 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944255+0000", 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016919328000000001 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.147 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944331+0000", 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6330999999999998e-05 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 107/66 ECSubWrite(tid=247, reqid=osd.7.0:247, at_version=107'100, trim_to=0'0, roll_forward_to=106'98))", 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1320545559999999, 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014333392, 2024-09-16T02:15:42.148 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.149 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929657+0000", 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7e-07 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929703+0000", 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6354999999999999e-05 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.150 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930028+0000", 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325528 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930050+0000", 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1165000000000001e-05 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930240+0000", 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000190729 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.151 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930267+0000", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6435e-05 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943883+0000", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013616453000000001 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943989+0000", 2024-09-16T02:15:42.152 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106257 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=129, reqid=client.4598.0:948, at_version=107'116, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1299608430000001, 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015009710000000001, 2024-09-16T02:15:42.153 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.154 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931752+0000", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0470000000000001e-06 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931799+0000", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6705000000000001e-05 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931979+0000", 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018057500000000001 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.155 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932002+0000", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2636000000000001e-05 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932166+0000", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016461500000000001 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932182+0000", 2024-09-16T02:15:42.156 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5536000000000002e-05 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946687+0000", 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014505120999999999 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946760+0000", 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2475e-05 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.157 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.bs5 107/83 ECSubWrite(tid=248, reqid=osd.7.0:248, at_version=107'73, trim_to=0'0, roll_forward_to=106'71))", 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1298265030000003, 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014855493000000001, 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:42.158 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931892+0000", 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9189999999999999e-06 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.159 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931934+0000", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1628999999999997e-05 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932094+0000", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159963 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932112+0000", 2024-09-16T02:15:42.160 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8168999999999998e-05 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.933629+0000", 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517357 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.933654+0000", 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4644999999999999e-05 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.161 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946650+0000", 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012995860999999999 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946740+0000", 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9950000000000004e-05 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.162 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=130, reqid=client.4598.0:949, at_version=107'117, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1235303739999996, 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011725154, 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:42.163 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938182+0000", 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.657e-06 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.164 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938232+0000", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0345e-05 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938407+0000", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017511499999999999 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938432+0000", 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4606000000000001e-05 2024-09-16T02:15:42.165 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944442+0000", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00600995 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944456+0000", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3814e-05 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949814+0000", 2024-09-16T02:15:42.166 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053581490000000004 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949905+0000", 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1518e-05 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=131, reqid=client.4598.0:950, at_version=107'118, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:42.167 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.1234693790000003, 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011753016, 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.168 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938242+0000", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.64e-07 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938301+0000", 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8535000000000001e-05 2024-09-16T02:15:42.169 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938489+0000", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018785399999999999 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938505+0000", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6626e-05 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.945894+0000", 2024-09-16T02:15:42.170 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0073882510000000002 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.945920+0000", 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6282999999999999e-05 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949914+0000", 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039943690000000002 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.171 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949994+0000", 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0134000000000002e-05 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.172 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:42.174 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:42.174 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:42.173+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:42.274 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:42.274+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:42.347 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:42.348 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:42.348 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:42.362 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:42.410 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:42.409+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:42.475 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:42.472+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:42.480 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:42.480 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:42.480 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:42.494 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:42.576 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:42.574+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:197 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback 2] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:42.664 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 75.338776577000004, 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013892593, 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 197 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.665 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322057+0000", 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322058+0000", 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.643e-06 2024-09-16T02:15:42.666 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322069+0000", 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.044e-05 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322090+0000", 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1455e-05 2024-09-16T02:15:42.667 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322106+0000", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6163000000000001e-05 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325364+0000", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032580809999999999 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.325616+0000", 2024-09-16T02:15:42.668 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025181900000000002 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.327103+0000", 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014866549999999999 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330072+0000", 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029688729999999999 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.669 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335817+0000", 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057450959999999999 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.335949+0000", 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000132368 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:198 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc 4=[4,3,2] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-16T02:15:42.670 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 75.338682184000007, 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014215244, 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 198 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.671 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322151+0000", 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322152+0000", 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3829999999999999e-06 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.672 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322159+0000", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.5660000000000003e-06 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322179+0000", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9613e-05 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.322190+0000", 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1542000000000001e-05 2024-09-16T02:15:42.673 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328227+0000", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0060372159999999998 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.328485+0000", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025810200000000002 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.329430+0000", 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00094416499999999996 2024-09-16T02:15:42.674 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.330786+0000", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013565509999999999 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336208+0000", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0054220529999999996 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:27.336366+0000", 2024-09-16T02:15:42.675 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015805300000000001 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:494 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.703770220999999, 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.020145202000000001, 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.676 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 494 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.677 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957063+0000", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957066+0000", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0340000000000001e-06 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957080+0000", 2024-09-16T02:15:42.678 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3696000000000001e-05 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957110+0000", 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0252000000000001e-05 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957135+0000", 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4586000000000001e-05 2024-09-16T02:15:42.679 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957642+0000", 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050762300000000002 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957870+0000", 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022741699999999999 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.680 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.960012+0000", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.002142793 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966293+0000", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062808969999999997 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977023+0000", 2024-09-16T02:15:42.681 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010729446 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.977208+0000", 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000185458 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:495 3.ds0 3:bad3968e:::smithi19530025-46:head [rollback d] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:42.682 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.703650443999997, 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021622549000000001, 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 495 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.683 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957183+0000", 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.684 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957185+0000", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7269999999999999e-06 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957191+0000", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.9029999999999998e-06 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957211+0000", 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0069e-05 2024-09-16T02:15:42.685 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957229+0000", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7856e-05 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961650+0000", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044206920000000004 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.961911+0000", 2024-09-16T02:15:42.686 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026131599999999999 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.963434+0000", 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00152287 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.966303+0000", 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0028691599999999999 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.687 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978597+0000", 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012293652 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978805+0000", 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000208304 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.688 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:496 3.ds0 3:bad3968e:::smithi19530025-46:head [append 0~0] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 55.703547919999998, 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021768414, 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 496 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.689 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957285+0000", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957287+0000", 2024-09-16T02:15:42.690 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.443e-06 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957294+0000", 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.7800000000000003e-06 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957313+0000", 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9669000000000001e-05 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.691 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.957328+0000", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4847e-05 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964542+0000", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0072143199999999998 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.964700+0000", 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157343 2024-09-16T02:15:42.692 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.965510+0000", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00080991199999999998 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.968798+0000", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0032880489999999999 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.978963+0000", 2024-09-16T02:15:42.693 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010165014 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:46.979054+0000", 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1037000000000003e-05 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s4 74/69 ECSubWrite(tid=137, reqid=osd.11.0:137, at_version=74'135, trim_to=0'0, roll_forward_to=73'131))", 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:42.694 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 39.680334361, 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.469769029, 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980499+0000", 2024-09-16T02:15:42.695 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980500+0000", 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5030000000000001e-06 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.980539+0000", 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.8754000000000002e-05 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.696 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448625+0000", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.46808555400000001 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448640+0000", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448730+0000", 2024-09-16T02:15:42.697 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9498000000000003e-05 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.448742+0000", 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2479e-05 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.450231+0000", 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014881320000000001 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.698 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:03.450268+0000", 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7307e-05 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:714 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 947105~871519,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.627397920999996, 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013949832000000001, 2024-09-16T02:15:42.699 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 714 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.700 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033435+0000", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033437+0000", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7129999999999999e-06 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033450+0000", 2024-09-16T02:15:42.701 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2782999999999999e-05 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033477+0000", 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7560999999999999e-05 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033493+0000", 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5133000000000001e-05 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.702 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.036275+0000", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0027821389999999999 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.036467+0000", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019283200000000001 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.047385+0000", 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010917671 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.703 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:715 3.ds0 3:ba16c3d6:::smithi19530025-37:head [read 328282~1490342,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 34.627240927000003, 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.01542255, 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.704 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 715 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.705 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033592+0000", 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033594+0000", 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.154e-06 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033608+0000", 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.706 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033644+0000", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5831999999999998e-05 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.033665+0000", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1121999999999999e-05 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041530+0000", 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078650109999999999 2024-09-16T02:15:42.707 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041711+0000", 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018134999999999999 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.049015+0000", 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0073036059999999998 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.708 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:770 3.ds0 3:ba16c3d6:::smithi19530025-37:head [append 0~491520] snapc 1b=[1b,1a,18,d] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 26.705514249, 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.021836291000000001, 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.709 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 770 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955319+0000", 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.710 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.955325+0000", 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.604e-06 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956113+0000", 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00078869400000000005 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.711 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956132+0000", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8702000000000002e-05 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956142+0000", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0456e-05 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956310+0000", 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000167467 2024-09-16T02:15:42.712 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.956490+0000", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018010199999999999 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.958995+0000", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0025054489999999999 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.962085+0000", 2024-09-16T02:15:42.713 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030896220000000002 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.977072+0000", 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014986963000000001 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:15.977155+0000", 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.3232000000000005e-05 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.714 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.ds0 89/32 ECSubWriteReply(tid=111, last_complete=89'92, committed=1, applied=1))", 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:18.482066+0000", 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 24.178766822, 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20899812400000001, 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.482066+0000", 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.715 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.482066+0000", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.482067+0000", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7399999999999991e-07 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.482076+0000", 2024-09-16T02:15:42.716 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6430000000000001e-06 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.690895+0000", 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.20881907199999999 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.690910+0000", 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5162e-05 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.717 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.691008+0000", 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8068999999999999e-05 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:18.691065+0000", 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6203999999999999e-05 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.718 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:923 3.ds0 3:ba16c3d6:::smithi19530025-37:head [copy-from ver 49] snapc 26=[26,25,24,22,21,20,1f,1e,1d,18] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:31.937633+0000", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.723199802, 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.043813484, 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 923 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.719 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937633+0000", 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937633+0000", 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937637+0000", 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1329999999999999e-06 2024-09-16T02:15:42.720 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937652+0000", 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5366000000000002e-05 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937684+0000", 2024-09-16T02:15:42.721 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.1644999999999999e-05 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937699+0000", 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5659999999999999e-05 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.722 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937850+0000", 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015066499999999999 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:15:42.723 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937863+0000", 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3040000000000001e-05 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.942739+0000", 2024-09-16T02:15:42.724 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00487632 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.942967+0000", 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022806700000000001 2024-09-16T02:15:42.725 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.974931+0000", 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.031963322000000002 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.726 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.980282+0000", 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053516919999999999 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.727 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.981201+0000", 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091860499999999996 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.981447+0000", 2024-09-16T02:15:42.728 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024596900000000002 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.729 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:924 3.ds0 3:ba16c3d6:::smithi19530025-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:31.937744+0000", 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 10.723089279, 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.042019053000000001, 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.730 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 924 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.731 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937744+0000", 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.732 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937744+0000", 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937745+0000", 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1030000000000001e-06 2024-09-16T02:15:42.733 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937750+0000", 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.2159999999999997e-06 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.734 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937769+0000", 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8674000000000001e-05 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.735 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.937778+0000", 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.3389999999999999e-06 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.938091+0000", 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031239700000000001 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.938115+0000", 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4426e-05 2024-09-16T02:15:42.736 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.943390+0000", 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0052749980000000004 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.943520+0000", 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000130183 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.737 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.979317+0000", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035796836999999998 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.979481+0000", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016398899999999999 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.979763+0000", 2024-09-16T02:15:42.738 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00028189099999999999 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:948 3.ds0 3:b3107852:::smithi19530025-9:head [append 0~0] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.923626+0000", 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.7372077050000003, 2024-09-16T02:15:42.739 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.032746373000000002, 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 948 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.740 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923626+0000", 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923626+0000", 2024-09-16T02:15:42.741 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923629+0000", 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.123e-06 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923643+0000", 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4253000000000001e-05 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.742 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923672+0000", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9436000000000001e-05 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923695+0000", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2999000000000001e-05 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927859+0000", 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0041638400000000002 2024-09-16T02:15:42.743 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.928132+0000", 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306299999999998 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930226+0000", 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020937320000000001 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.744 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.939626+0000", 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094002310000000002 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.956181+0000", 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016554620999999999 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.745 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.956372+0000", 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191075 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:949 3.ds0 3:b3107852:::smithi19530025-9:head [rollback 21] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.923736+0000", 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.7370977249999999, 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.035050855999999998, 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.746 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 949 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923736+0000", 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.747 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923736+0000", 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923737+0000", 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4130000000000001e-06 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.748 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923745+0000", 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1529999999999997e-06 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923767+0000", 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1603e-05 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923782+0000", 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.573e-05 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.749 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931581+0000", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0077982520000000003 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931941+0000", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00035995599999999998 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.933417+0000", 2024-09-16T02:15:42.750 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001476492 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.939637+0000", 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062201310000000003 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.958641+0000", 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019003460999999999 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.751 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.958786+0000", 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00014566500000000001 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:950 3.ds0 3:b3107852:::smithi19530025-9:head [append 0~0] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.923808+0000", 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.7370247699999997, 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.038042016999999997, 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.752 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 950 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923808+0000", 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.753 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923808+0000", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923810+0000", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.263e-06 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923819+0000", 2024-09-16T02:15:42.754 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.7830000000000004e-06 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923838+0000", 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9193000000000001e-05 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.923854+0000", 2024-09-16T02:15:42.755 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6322999999999999e-05 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.936238+0000", 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012384077 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.936497+0000", 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00025883600000000003 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.756 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.937829+0000", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013324649999999999 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943229+0000", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053996820000000003 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.961651+0000", 2024-09-16T02:15:42.757 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018422291 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.961850+0000", 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000199104 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:958 3.ds0 3:ba16c3d6:::smithi19530025-37:head [copy-from ver 0] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.924621+0000", 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.736212407, 2024-09-16T02:15:42.758 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017394517000000002, 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 958 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924621+0000", 2024-09-16T02:15:42.759 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924621+0000", 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924623+0000", 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3860000000000001e-06 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.760 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924650+0000", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6460000000000001e-05 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924719+0000", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8944999999999998e-05 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924742+0000", 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3699000000000001e-05 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.761 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938857+0000", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014115008 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.939049+0000", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00019190400000000001 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.942015+0000", 2024-09-16T02:15:42.762 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0029661150000000001 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:959 3.ds0 3:ba16c3d6:::smithi19530025-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.924779+0000", 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 6.7360541019999998, 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018455223999999999, 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:42.763 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 959 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924779+0000", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924779+0000", 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.764 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924781+0000", 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7740000000000001e-06 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924790+0000", 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8859999999999993e-06 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.765 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924805+0000", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5432999999999999e-05 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924816+0000", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0273e-05 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.939460+0000", 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014644959000000001 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.766 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.939592+0000", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013171 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.942630+0000", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0030377220000000001 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.942868+0000", 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000238397 2024-09-16T02:15:42.767 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943234+0000", 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00036607000000000002 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 109/83 ECSubWrite(tid=386, reqid=osd.13.0:386, at_version=109'165, trim_to=0'0, roll_forward_to=109'161))", 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:37.982956+0000", 2024-09-16T02:15:42.768 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.6778773679999999, 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.058020003000000001, 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.982956+0000", 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.769 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.982956+0000", 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.982962+0000", 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9240000000000004e-06 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.982999+0000", 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7202000000000001e-05 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.770 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.038828+0000", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.055828994999999999 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.038915+0000", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.6867999999999996e-05 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.039085+0000", 2024-09-16T02:15:42.771 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000169801 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.039113+0000", 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.883e-05 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.040873+0000", 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017596319999999999 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.772 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.040976+0000", 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010275100000000001 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 109/83 ECSubWrite(tid=389, reqid=osd.13.0:389, at_version=109'169, trim_to=0'0, roll_forward_to=109'166))", 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:38.046376+0000", 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.6144570590000002, 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.115734881, 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.773 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.046376+0000", 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.046376+0000", 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.774 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.046378+0000", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3400000000000001e-06 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.046413+0000", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.5924999999999999e-05 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.160190+0000", 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.11377619 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.775 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.160205+0000", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.535e-05 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.160358+0000", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015348299999999999 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.160388+0000", 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.9771999999999999e-05 2024-09-16T02:15:42.776 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.162032+0000", 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00164358 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.162111+0000", 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9240999999999996e-05 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.777 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s2 109/83 ECSubWrite(tid=393, reqid=client.4598.0:984, at_version=0'0, trim_to=0'0, roll_forward_to=109'172))", 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:38.190985+0000", 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 4.4698478389999998, 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.128890852, 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.190985+0000", 2024-09-16T02:15:42.778 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.190985+0000", 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.190987+0000", 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9190000000000002e-06 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.779 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.191008+0000", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0400000000000001e-05 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.318784+0000", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.12777617199999999 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.318799+0000", 2024-09-16T02:15:42.780 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5004999999999999e-05 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.318914+0000", 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114901 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.318929+0000", 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.535e-05 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.781 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.319799+0000", 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00087031000000000001 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.319876+0000", 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6794999999999999e-05 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:42.782 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:42.783 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:42.784 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:42.785 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:42.781+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:42.884 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:42.882+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:42.913 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:42.913 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:42.913 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:42.913 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:42.927 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:42.986 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:42.981+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:43.011 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-09-16T02:15:43.051 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:43.051 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:43.051 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:43.051 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:43.051 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:43.065 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:43.087 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:43.086+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:43.187 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:43.185+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:43.194 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:43.194 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:43.195 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:43.195 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:43.195 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.195 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 35/32 ECSubWrite(tid=114, reqid=client.4598.0:28, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.195 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.898066502999995, 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014358203, 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.196 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.293965+0000", 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.197 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294123+0000", 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000157742 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294233+0000", 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000109842 2024-09-16T02:15:43.198 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294256+0000", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3779000000000001e-05 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294602+0000", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000345817 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.294625+0000", 2024-09-16T02:15:43.199 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3071999999999999e-05 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308228+0000", 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013602725 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.308323+0000", 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4631999999999998e-05 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.200 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.894130236999999, 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013936119, 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.201 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.297901+0000", 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7299999999999996e-07 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.202 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298091+0000", 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000189612 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298361+0000", 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00026983699999999998 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.203 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298377+0000", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6619000000000001e-05 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298580+0000", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020297699999999999 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298602+0000", 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1823000000000001e-05 2024-09-16T02:15:43.204 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311758+0000", 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013155635000000001 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311837+0000", 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9042999999999995e-05 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.205 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.893933129000004, 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013796841000000001, 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.206 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298098+0000", 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298099+0000", 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2999999999999999e-07 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.207 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298132+0000", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3460999999999997e-05 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298437+0000", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000305149 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.298455+0000", 2024-09-16T02:15:43.208 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8099000000000001e-05 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299840+0000", 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001384826 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5151999999999999e-05 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.209 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311844+0000", 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011978284000000001 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.311895+0000", 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1039999999999999e-05 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.210 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.890604459000002, 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013049734, 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.211 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.212 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301427+0000", 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8700000000000004e-07 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.213 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6052000000000001e-05 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301694+0000", 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000240423 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301716+0000", 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2289e-05 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.214 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305459+0000", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0037425710000000001 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.305488+0000", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8756e-05 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314378+0000", 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0088900869999999996 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.215 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.314476+0000", 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8669e-05 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:43.216 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.890218322999999, 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014210883000000001, 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.217 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301813+0000", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.301814+0000", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4130000000000001e-06 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302022+0000", 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000207844 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.218 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302120+0000", 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.7864999999999999e-05 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302140+0000", 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0156000000000001e-05 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.219 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310588+0000", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0084482120000000001 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.310612+0000", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3879e-05 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315921+0000", 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053087079999999997 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.220 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316024+0000", 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102806 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.1s1 35/32 ECSubWrite(tid=5, reqid=client.4598.0:41, at_version=35'6, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:43.221 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.889206943999994, 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013263980999999999, 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.222 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302824+0000", 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302826+0000", 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9240000000000001e-06 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.223 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.302901+0000", 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5172999999999993e-05 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304024+0000", 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001123181 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.304072+0000", 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8127999999999997e-05 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.224 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312045+0000", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079722669999999999 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.312071+0000", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6291999999999998e-05 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316033+0000", 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039623380000000001 2024-09-16T02:15:43.225 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316088+0000", 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.4678000000000001e-05 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:49 3.cs0 3:3203a6be:::smithi19530025-17:head [append 0~278528,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.876258911999997, 2024-09-16T02:15:43.226 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016954778, 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 49 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:43.227 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315772+0000", 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.315781+0000", 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.8699999999999998e-06 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.228 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316702+0000", 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00092113000000000004 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316734+0000", 2024-09-16T02:15:43.229 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2375000000000001e-05 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316756+0000", 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1843000000000001e-05 2024-09-16T02:15:43.230 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317070+0000", 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031399899999999997 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.317385+0000", 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00031517099999999999 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.231 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.320926+0000", 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035407450000000001 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332340+0000", 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011414260000000001 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332532+0000", 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191137 2024-09-16T02:15:43.232 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332727+0000", 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000195248 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:50 3.cs0 3:3203a6be:::smithi19530025-17:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.875218129999993, 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017275975999999998, 2024-09-16T02:15:43.233 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 50 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.234 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316813+0000", 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316815+0000", 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0899999999999999e-06 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.235 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316826+0000", 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1242e-05 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316851+0000", 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4969000000000001e-05 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316868+0000", 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6535999999999999e-05 2024-09-16T02:15:43.236 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324308+0000", 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0074406289999999998 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.324542+0000", 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023414200000000001 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.237 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.325642+0000", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001099182 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.332739+0000", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0070972090000000002 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.333048+0000", 2024-09-16T02:15:43.238 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00030949200000000002 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334089+0000", 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010404850000000001 2024-09-16T02:15:43.239 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:51 3.cs0 3:3203a6be:::smithi19530025-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:43.240 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 78.875132633999996, 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019761212, 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 51 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.241 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316898+0000", 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.242 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316900+0000", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.477e-06 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316907+0000", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8889999999999996e-06 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316927+0000", 2024-09-16T02:15:43.243 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0591999999999999e-05 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.316943+0000", 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5756999999999998e-05 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327066+0000", 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010123037 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.244 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.327269+0000", 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00020309000000000001 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.245 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334254+0000", 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0069844 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.246 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.334498+0000", 2024-09-16T02:15:43.247 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00024477499999999997 2024-09-16T02:15:43.247 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.247 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.247 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.247 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:24.336660+0000", 2024-09-16T02:15:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0021611949999999999 2024-09-16T02:15:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.248 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 51/32 ECSubWrite(tid=224, reqid=osd.13.0:224, at_version=51'45, trim_to=0'0, roll_forward_to=49'43))", 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 62.777116112000002, 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24539247, 2024-09-16T02:15:43.249 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.250 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:43.251 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414915+0000", 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2000000000000001e-07 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.252 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.253 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.253 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.414963+0000", 2024-09-16T02:15:43.253 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8265e-05 2024-09-16T02:15:43.253 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.253 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658566+0000", 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.24360277699999999 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.254 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658581+0000", 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4253000000000001e-05 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658749+0000", 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000168691 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.255 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.658768+0000", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8332999999999999e-05 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660236+0000", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0014682129999999999 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.660307+0000", 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.1618000000000004e-05 2024-09-16T02:15:43.256 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWriteReply(3.1s1 62/61 ECSubWriteReply(tid=22, last_complete=62'113, committed=1, applied=1))", 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 52.16046746, 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66094662900000001, 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.257 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031564+0000", 2024-09-16T02:15:43.258 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.2300000000000002e-07 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.031567+0000", 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.3830000000000001e-06 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690362+0000", 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.65879452900000002 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.259 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690376+0000", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.397e-05 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.690498+0000", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012224 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.692510+0000", 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020120839999999999 2024-09-16T02:15:43.260 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:525 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.500449042, 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014305174, 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:43.261 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 525 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:43.262 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691582+0000", 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.263 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691585+0000", 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.667e-06 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691596+0000", 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1837e-05 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.264 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691625+0000", 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8714999999999999e-05 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691640+0000", 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5027e-05 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.265 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.266 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.266 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.693656+0000", 2024-09-16T02:15:43.266 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0020154190000000001 2024-09-16T02:15:43.266 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.266 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.694058+0000", 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00040255 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.267 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.268 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.705887+0000", 2024-09-16T02:15:43.268 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011828959 2024-09-16T02:15:43.268 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.268 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.268 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:526 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.500363084, 2024-09-16T02:15:43.269 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.018909656, 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 526 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.270 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.271 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691668+0000", 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691669+0000", 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.19e-06 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.272 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691676+0000", 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.277e-06 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691700+0000", 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3842e-05 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.273 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691718+0000", 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.823e-05 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695237+0000", 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0035182619999999999 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.274 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.695450+0000", 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021319899999999999 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.710578+0000", 2024-09-16T02:15:43.275 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015127656 2024-09-16T02:15:43.276 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.276 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.276 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.276 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.276 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:527 3.1s1 3:8a412445:::smithi19530025-48:head [read 0~1572864,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.500287688999997, 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.023766461999999999, 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.277 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 527 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.278 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691743+0000", 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691744+0000", 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.2299999999999999e-07 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.279 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691749+0000", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0370000000000002e-06 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691763+0000", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.342e-05 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.691770+0000", 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.5320000000000001e-06 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.280 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.697667+0000", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0058967209999999997 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.698047+0000", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000379768 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715510+0000", 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017463060999999998 2024-09-16T02:15:43.281 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpReadReply(3.1s1 62/61 ECSubReadReply(tid=47, attrs_read=0))", 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.489697628000002, 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013250526, 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.282 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702333+0000", 2024-09-16T02:15:43.283 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702334+0000", 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.074e-06 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702822+0000", 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00048715800000000001 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.284 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702872+0000", 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0565000000000001e-05 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.702881+0000", 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.49e-06 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.711121+0000", 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082399199999999995 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.285 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.715584+0000", 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0044633190000000003 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDPGPushReply(3.1s1 62/61 [PushReplyOp(3:830a71a8:::smithi19530025-14:head)])", 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 51.487817474000003, 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19260793200000001, 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.286 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.287 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704214+0000", 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1399999999999997e-07 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.704220+0000", 2024-09-16T02:15:43.288 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.1759999999999998e-06 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894917+0000", 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.19069640900000001 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.894932+0000", 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5690000000000001e-05 2024-09-16T02:15:43.289 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.895110+0000", 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000177546 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:51.896821+0000", 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0017116970000000001 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.290 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 73/32 ECSubWrite(tid=272, reqid=osd.13.0:272, at_version=73'82, trim_to=0'0, roll_forward_to=72'78))", 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 41.237599428999999, 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35829697500000002, 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:43.291 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954432+0000", 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954434+0000", 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.933e-06 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.292 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:01.954491+0000", 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7326e-05 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310497+0000", 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.35600593200000002 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.293 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310519+0000", 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2121999999999999e-05 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310700+0000", 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018157399999999999 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.294 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.310716+0000", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5696e-05 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.312660+0000", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001944274 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:02.312729+0000", 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8118000000000001e-05 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.295 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.2s4 109/83 ECSubWrite(tid=381, reqid=client.4598.0:968, at_version=109'161, trim_to=0'0, roll_forward_to=106'159))", 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:37.934518+0000", 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.2575130479999999, 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019934640999999999, 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.296 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934518+0000", 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934518+0000", 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934518+0000", 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.0699999999999998e-07 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.297 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934558+0000", 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.9412000000000001e-05 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934667+0000", 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010912399999999999 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.298 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.934689+0000", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2535999999999999e-05 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.935021+0000", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033134400000000002 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.935048+0000", 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6956000000000001e-05 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.299 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.954399+0000", 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.019350926000000001 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.954453+0000", 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3936e-05 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.300 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.5s3 109/83 ECSubWrite(tid=61, reqid=osd.12.0:61, at_version=109'74, trim_to=0'0, roll_forward_to=108'72))", 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:37.939235+0000", 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.2527958740000003, 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011711813999999999, 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939235+0000", 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.301 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939235+0000", 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939236+0000", 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.302 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939293+0000", 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6845999999999997e-05 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939394+0000", 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010139700000000001 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939418+0000", 2024-09-16T02:15:43.303 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4102999999999999e-05 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939684+0000", 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000265476 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.939721+0000", 2024-09-16T02:15:43.304 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.7462000000000001e-05 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.950847+0000", 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011126270000000001 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.950947+0000", 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.9684000000000003e-05 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.305 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "replica_scrub(pg: 3.7s6,from:0'0,to:108'144,epoch:109/32,start:3:e0000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:37.958664+0000", 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 5.2333672599999996, 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33671040400000002, 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.306 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.958664+0000", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.958664+0000", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.958666+0000", 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.9259999999999999e-06 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.307 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:37.958680+0000", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3934000000000001e-05 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.295102+0000", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.33642283699999997 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.295118+0000", 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5707000000000001e-05 2024-09-16T02:15:43.308 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.295291+0000", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000172788 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.295318+0000", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.7076999999999998e-05 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:38.295374+0000", 2024-09-16T02:15:43.309 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.6134999999999997e-05 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:43.310 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:43.312 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:43.312 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:43.309+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:43.412 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:43.408+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:43.446 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:43.446 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:43.446 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:43.446 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:43.461 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:43.513 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:43.509+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:43.583 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:43.583 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:43.583 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:43.583 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:43.583 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:43.597 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:43.613 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:43.610+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:43.714 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:43.709+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 38/32 ECSubWrite(tid=16, reqid=client.4598.0:198, at_version=0'0, trim_to=0'0, roll_forward_to=38'10))", 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 76.379419353000003, 2024-09-16T02:15:43.719 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.334730205, 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.720 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335871+0000", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335879+0000", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-06 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.335907+0000", 2024-09-16T02:15:43.721 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7291999999999999e-05 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669030+0000", 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.33312349400000002 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669045+0000", 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4943000000000001e-05 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.722 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669245+0000", 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019942899999999999 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.669263+0000", 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8445999999999999e-05 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670533+0000", 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001270024 2024-09-16T02:15:43.723 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:27.670602+0000", 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8406999999999997e-05 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 44/32 ECSubWrite(tid=22, reqid=osd.2.0:22, at_version=44'14, trim_to=0'0, roll_forward_to=43'12))", 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:43.724 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 70.380446547000005, 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75881597999999995, 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.725 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334844+0000", 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334845+0000", 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.0900000000000001e-07 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.726 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:33.334886+0000", 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1254999999999997e-05 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.727 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091945+0000", 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.75705888700000001 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.091960+0000", 2024-09-16T02:15:43.728 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4686e-05 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092157+0000", 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00019717000000000001 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.729 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.092176+0000", 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9020000000000001e-05 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093578+0000", 2024-09-16T02:15:43.730 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001401769 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.093660+0000", 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2484000000000005e-05 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.731 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.fs6 45/32 ECSubWrite(tid=24, reqid=osd.2.0:24, at_version=45'16, trim_to=0'0, roll_forward_to=44'14))", 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 69.363423448999995, 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017058792, 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.732 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351867+0000", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351868+0000", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3e-07 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.351917+0000", 2024-09-16T02:15:43.733 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8987999999999999e-05 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352051+0000", 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013457599999999999 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352073+0000", 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1489e-05 2024-09-16T02:15:43.734 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352231+0000", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000158455 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.352260+0000", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8472000000000001e-05 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368854+0000", 2024-09-16T02:15:43.735 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016594647000000001 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.368926+0000", 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1534999999999999e-05 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 45/32 ECSubWriteReply(tid=35, last_complete=45'32, committed=1, applied=1))", 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:43.736 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 69.333509425000003, 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65850650099999997, 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.737 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381781+0000", 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381783+0000", 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.857e-06 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.381795+0000", 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1466e-05 2024-09-16T02:15:43.738 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040081+0000", 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.65828677099999999 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040095+0000", 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3686000000000001e-05 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.739 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040215+0000", 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00011949 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.040288+0000", 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3231000000000004e-05 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.740 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s1 45/32 ECSubWrite(tid=25, reqid=client.4598.0:255, at_version=45'16, trim_to=0'0, roll_forward_to=45'14))", 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 69.318303331999999, 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27792197099999999, 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.741 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396987+0000", 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.742 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.396988+0000", 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0759999999999999e-06 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.397010+0000", 2024-09-16T02:15:43.743 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1525000000000001e-05 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673630+0000", 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.27662027 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.744 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673655+0000", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5003000000000001e-05 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673857+0000", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020175299999999999 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.673875+0000", 2024-09-16T02:15:43.745 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8283000000000001e-05 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674832+0000", 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00095631999999999998 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:34.674909+0000", 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7741e-05 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.746 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:487 3.7s0 3:ed091bc8:::smithi19530025-23:head [create,setxattr gochgegikXngWahUkkjRl^QTgYhb[b]WXamd_eSq (2384) in=40b,setxattr b\\PtoYR`hmWtnd_a[S\\\\lXhWq]eThtkqjphngsWR (1473) in=40b,setxattr Ub[Vfm^UiUZrRqc[oNaY`rgk\\rgTkNpothWnk[oU (771) in=40b,setxattr OOq^bpU^bm (1437) in=10b,setxattr biYdnofqNd_eZg\\amb``Na_saZ^ (1437) in=27b,setxattr rX`YVl[kmaRi[QfikjQasTjWZSRiWrW`bkcandUf (35) in=40b,setxattr [YUqn^[Zb^O\\a_bmaVfNcegQesrbqaZSZoip[Xdr (5752) in=40b,setxattr fceY\\cr[bnaj[ (1437) in=13b,setxattr acNNOX`hScRmkbQpOdbgSnbYo_aTYX^]SiYpcXhV (1435) in=40b,setxattr olt]ab (1437) in=6b,setxattr UlYSXgbei[Yp^prXjdVWknUpaYc`bdacfOTgrmOe (279) in=40b,setxattr Pe[lpnRflfpehZ[\\\\niYcpg]NTejdgUaQ\\^PY[pQ (1122) in=40b,setxattr \\oimfXol\\dqgos_Rkh\\U`blYV\\`qdZco_d`fnl\\a (877) in=40b,setxattr ahhrfeWh_qaa_ohbtahY_RbYoQU`Y`R[bTOOhsgS (593) in=40b,setxattr iUbWaVpfdgVcRsPQUtUOtQdYdVWantYNjbgOPU^t (842) in=40b,setxattr \\R^[`cSkXTlebhQo^[nNOpilmsejaimZd[tgWYb_ (1438) in=40b,setxattr PbnTj^iPVhr`VaY`i[oQ[Pj]b[ZjaaSgYfVfWhrX (2805) in=40b,setxattr NSphstUlmcX]Uhs]jPZ^nXY]Rh^oVkSZfn`[OQRZ (1542) in=40b,setxattr Ysnsrf[rY`mflrhPbnkV_\\dRrQPeld[oXqV[Stcn (842) in=40b,setxattr \\rht]cXhNbU__TUnbohctXit^XTsnoTUNTPNY[fp (1962) in=40b] snapc d=[d,a] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.782677331000002, 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019918874, 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 487 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.747 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932613+0000", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932617+0000", 2024-09-16T02:15:43.748 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.309e-06 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932695+0000", 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8548999999999996e-05 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932736+0000", 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0349000000000003e-05 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.749 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932757+0000", 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1903e-05 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.750 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932928+0000", 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00017018600000000001 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.932950+0000", 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2395000000000002e-05 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934568+0000", 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00161812 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.751 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.934773+0000", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000204995 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.936578+0000", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001804709 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.940066+0000", 2024-09-16T02:15:43.752 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034883869999999999 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952401+0000", 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012334539 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.952532+0000", 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000131433 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.753 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 57/32 ECSubWrite(tid=48, reqid=osd.6.0:48, at_version=57'40, trim_to=0'0, roll_forward_to=55'38))", 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 57.774275555000003, 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013412999, 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.754 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941015+0000", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941016+0000", 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6000000000000004e-07 2024-09-16T02:15:43.755 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941065+0000", 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9462000000000002e-05 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941113+0000", 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.756 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941128+0000", 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4348999999999999e-05 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941254+0000", 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012600400000000001 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.941277+0000", 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3482999999999999e-05 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.757 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954333+0000", 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013055430999999999 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:45.954428+0000", 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5468000000000001e-05 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.758 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=86, reqid=osd.6.0:86, at_version=73'70, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.770042967000002, 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015494396000000001, 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.759 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945248+0000", 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945254+0000", 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.3130000000000004e-06 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.760 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945315+0000", 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.0584999999999997e-05 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945408+0000", 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.3288999999999994e-05 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.761 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945425+0000", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7329e-05 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945587+0000", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016137399999999999 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.945615+0000", 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8048999999999999e-05 2024-09-16T02:15:43.762 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959667+0000", 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014052007 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.960742+0000", 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010754499999999999 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.763 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 73/32 ECSubWrite(tid=87, reqid=osd.6.0:87, at_version=73'72, trim_to=0'0, roll_forward_to=70'68))", 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.767213101000003, 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020784545000000001, 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.764 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948078+0000", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948079+0000", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.877e-06 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948132+0000", 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2312000000000003e-05 2024-09-16T02:15:43.765 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948237+0000", 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000105015 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948259+0000", 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2668999999999999e-05 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.766 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948410+0000", 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015014100000000001 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.948436+0000", 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6060000000000001e-05 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968776+0000", 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.020339946000000001 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.767 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.968862+0000", 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6525000000000003e-05 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.cs5 73/32 ECSubWrite(tid=80, reqid=osd.14.0:80, at_version=73'29, trim_to=0'0, roll_forward_to=70'27))", 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.752570829, 2024-09-16T02:15:43.768 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019480810000000001, 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962720+0000", 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.769 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962721+0000", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1699999999999997e-07 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962770+0000", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.9233000000000001e-05 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962872+0000", 2024-09-16T02:15:43.770 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010231800000000001 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.962901+0000", 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8765999999999999e-05 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.771 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963118+0000", 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00021719600000000001 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.963156+0000", 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7435999999999999e-05 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982116+0000", 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018960226 2024-09-16T02:15:43.772 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982201+0000", 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4818e-05 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=133, reqid=osd.11.0:133, at_version=73'129, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.749495408000001, 2024-09-16T02:15:43.773 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016499698, 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:43.774 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.775 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965795+0000", 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.17e-07 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.776 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965821+0000", 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.569e-05 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965896+0000", 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.5185999999999995e-05 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.777 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965907+0000", 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0379e-05 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966895+0000", 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00098857000000000003 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.966922+0000", 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7066000000000001e-05 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.778 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982221+0000", 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.015298589 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982295+0000", 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4000999999999998e-05 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.1s6 73/69 ECSubWrite(tid=134, reqid=osd.11.0:134, at_version=73'131, trim_to=0'0, roll_forward_to=71'127))", 2024-09-16T02:15:43.779 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 41.749467473999999, 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017078139999999999, 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.780 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965823+0000", 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.03e-07 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.781 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965841+0000", 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7326e-05 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965939+0000", 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.8420999999999998e-05 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.782 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.965956+0000", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6988999999999999e-05 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972222+0000", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062658890000000002 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.972246+0000", 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4403e-05 2024-09-16T02:15:43.783 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982860+0000", 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010613734 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.982901+0000", 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1174999999999998e-05 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.784 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(osd.6.22:4 3.7s0 3:ed091bc8:::smithi19530025-23:head [assert-version v49,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 11.763456135, 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017925606, 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.785 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951835+0000", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951838+0000", 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7000000000000002e-06 2024-09-16T02:15:43.786 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951854+0000", 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5563e-05 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951902+0000", 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8402000000000002e-05 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.787 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.951924+0000", 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1469000000000001e-05 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.952770+0000", 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00084661899999999995 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.953027+0000", 2024-09-16T02:15:43.788 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00025700000000000001 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:31.969760+0000", 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016732852999999999 2024-09-16T02:15:43.789 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 106/32 ECSubWriteReply(tid=138, last_complete=106'134, committed=1, applied=1))", 2024-09-16T02:15:43.790 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.053950+0000", 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 8.6613406269999995, 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88552325399999998, 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.791 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.053950+0000", 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.792 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.053950+0000", 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.053951+0000", 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.44e-07 2024-09-16T02:15:43.793 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.053955+0000", 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.6129999999999997e-06 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.794 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.938096+0000", 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88414066000000002 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.795 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.938104+0000", 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7000000000000008e-06 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.938287+0000", 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000183747 2024-09-16T02:15:43.796 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.939473+0000", 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00118589 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:951 3.7s0 3:ed091bc8:::smithi19530025-23:head [append 0~0] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.922886+0000", 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.7924050229999997, 2024-09-16T02:15:43.797 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.022655057999999999, 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 951 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922886+0000", 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.798 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922886+0000", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922888+0000", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3300000000000001e-06 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922898+0000", 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6809999999999993e-06 2024-09-16T02:15:43.799 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922919+0000", 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1471000000000002e-05 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922933+0000", 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3577999999999999e-05 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.800 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923011+0000", 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8285e-05 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923019+0000", 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.4349999999999997e-06 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.926432+0000", 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0034128969999999998 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.801 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.926578+0000", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014527499999999999 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.927486+0000", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00090879899999999996 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.933896+0000", 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0064098219999999999 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.802 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.945472+0000", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011575818999999999 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.945541+0000", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.8665999999999996e-05 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:952 3.7s0 3:ed091bc8:::smithi19530025-23:head [rollback 1e] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.922958+0000", 2024-09-16T02:15:43.803 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.7923330430000002, 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024764085000000002, 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 952 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.804 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922958+0000", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922958+0000", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922958+0000", 2024-09-16T02:15:43.805 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.9800000000000003e-07 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922961+0000", 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.27e-06 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922971+0000", 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1770000000000004e-06 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.806 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.922979+0000", 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6000000000000007e-06 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923076+0000", 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.6807999999999999e-05 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923080+0000", 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.0470000000000004e-06 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.807 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.928139+0000", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0050586670000000002 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.928241+0000", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000102503 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.932021+0000", 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037796370000000002 2024-09-16T02:15:43.808 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.935795+0000", 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037745550000000002 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.947656+0000", 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011860556 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.809 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.947722+0000", 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5667000000000007e-05 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:953 3.7s0 3:ed091bc8:::smithi19530025-23:head [append 0~0] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.923002+0000", 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.7922889230000001, 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024912858, 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:43.810 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 953 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923002+0000", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923002+0000", 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.811 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923002+0000", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.6199999999999998e-07 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923006+0000", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.405e-06 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923016+0000", 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.9170000000000001e-06 2024-09-16T02:15:43.812 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923023+0000", 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.041e-06 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923097+0000", 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4559000000000002e-05 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.813 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.923100+0000", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.224e-06 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.932928+0000", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0098276519999999992 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.933063+0000", 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000135043 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.814 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.933477+0000", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00041382000000000002 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.935799+0000", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002321816 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.947849+0000", 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.012050171 2024-09-16T02:15:43.815 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.947915+0000", 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5648000000000006e-05 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.7s0 107/32 ECSubWriteReply(tid=142, last_complete=107'140, committed=1, applied=1))", 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.944251+0000", 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.7710394279999999, 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.053785434, 2024-09-16T02:15:43.816 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.944251+0000", 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.944251+0000", 2024-09-16T02:15:43.817 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.944252+0000", 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7300000000000002e-07 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.944254+0000", 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.5770000000000001e-06 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.818 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.996883+0000", 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.052628335999999998 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.996898+0000", 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5860000000000001e-05 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.996963+0000", 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4917000000000002e-05 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.819 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.998037+0000", 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001073371 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.820 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds6 107/32 ECSubWrite(tid=132, reqid=client.4598.0:950, at_version=0'0, trim_to=0'0, roll_forward_to=107'118))", 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:35.961664+0000", 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 7.753626508, 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94618140100000003, 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.961664+0000", 2024-09-16T02:15:43.821 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.961664+0000", 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.822 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.961666+0000", 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6109999999999999e-06 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:35.961680+0000", 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4569000000000001e-05 2024-09-16T02:15:43.823 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.906381+0000", 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.94470097799999997 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.824 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.906400+0000", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9035999999999999e-05 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.906534+0000", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00013332099999999999 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.906553+0000", 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.9236e-05 2024-09-16T02:15:43.825 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.907771+0000", 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012181519999999999 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:36.907846+0000", 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4498000000000004e-05 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.826 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDScrubReserve(3.7s0,GRANT,e:108,from:14(6),reservation_nonce:0)", 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:37.755396+0000", 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 5.9598950080000002, 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.193308705, 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.755396+0000", 2024-09-16T02:15:43.827 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.755396+0000", 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.755398+0000", 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.917e-06 2024-09-16T02:15:43.828 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.755407+0000", 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.5570000000000001e-06 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.948051+0000", 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19264347900000001 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:43.829 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.948066+0000", 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5356e-05 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.948601+0000", 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00053493200000000001 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.948631+0000", 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9989e-05 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:43.830 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:37.948704+0000", 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3474999999999997e-05 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:43.831 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:43.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:43.833 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:43.834 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:43.828+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:43.933 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:43.929+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:44.002 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:44.002 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:44.004 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:44.004 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:44.004 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:44.004 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:44.018 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 11] out_osds: [10] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-16T02:15:44.018 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:44.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:15:44.019 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:44.030 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:44.030 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:44.056 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:44.056 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:44.064 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:44.061+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:44.081 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:44.081 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:44.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:44.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:44.135 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:44.135 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:44.129+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:44.165 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:44.165 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:44.190 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:44.190 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:44.190 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:44.191 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:44.205 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:44.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:44.211 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:44.236 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:44.236 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:44.248 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:44.245+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:44.263 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:44.264 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:44.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:44.289 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:44.315 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:44.315 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:44.330 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:15:44.331 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:15:44.337 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:44.336+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:44.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:44.345 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:44.347 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-16T02:15:44.022753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114229811,"num_objects":99,"num_object_clones":48,"num_object_copies":673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":6,"num_read":538,"num_read_kb":159792,"num_write":1080,"num_write_kb":156178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1499,"ondisk_log_size":1499,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":678520,"kb_used_data":210676,"kb_used_omap":23,"kb_used_meta":467496,"kb_avail":1414899080,"statfs":{"total":1449551462400,"available":1448856657920,"internally_reserved":0,"allocated":215732224,"data_stored":286570280,"data_compressed":3990499,"data_compressed_allocated":5480448,"data_compressed_original":11821056,"omap_allocated":23922,"internal_metadata":478716558},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1818624,"num_objects":11,"num_object_clones":11,"num_object_copies":77,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":29,"num_read_kb":28123,"num_write":63,"num_write_kb":688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002087"},"pg_stats":[{"pgid":"3.f","version":"110'55","reported_seq":437,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.987639+0000","last_change":"2024-09-16T02:15:39.026154+0000","last_active":"2024-09-16T02:15:39.987639+0000","last_peered":"2024-09-16T02:15:39.987639+0000","last_clean":"2024-09-16T02:15:39.987639+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:15:39.987639+0000","last_undegraded":"2024-09-16T02:15:39.987639+0000","last_fullsized":"2024-09-16T02:15:39.987639+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.420465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020148758999999999,"stat_sum":{"num_bytes":4751360,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":2907,"num_write":38,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.e","version":"106'79","reported_seq":549,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.033992+0000","last_change":"2024-09-16T02:15:38.989889+0000","last_active":"2024-09-16T02:15:39.033992+0000","last_peered":"2024-09-16T02:15:39.033992+0000","last_clean":"2024-09-16T02:15:39.033992+0000","last_became_active":"2024-09-16T02:15:13.070271+0000","last_became_peered":"2024-09-16T02:15:13.070271+0000","last_unstale":"2024-09-16T02:15:39.033992+0000","last_undegraded":"2024-09-16T02:15:39.033992+0000","last_fullsized":"2024-09-16T02:15:39.033992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'66","last_scrub_stamp":"2024-09-16T02:15:27.680026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.680026+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.051994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00059237999999999999,"stat_sum":{"num_bytes":11616256,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":18552,"num_write":73,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,14,11,0,15],"acting":[8,9,1,14,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.d","version":"110'138","reported_seq":505,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.874295+0000","last_change":"2024-09-16T02:15:39.027280+0000","last_active":"2024-09-16T02:15:39.874295+0000","last_peered":"2024-09-16T02:15:39.874295+0000","last_clean":"2024-09-16T02:15:39.874295+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:15:39.874295+0000","last_undegraded":"2024-09-16T02:15:39.874295+0000","last_fullsized":"2024-09-16T02:15:39.874295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.027258724000000002,"stat_sum":{"num_bytes":6602752,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":15024,"num_write":62,"num_write_kb":12320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.c","version":"99'31","reported_seq":411,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.986081+0000","last_change":"2024-09-16T02:15:38.986870+0000","last_active":"2024-09-16T02:15:38.986081+0000","last_peered":"2024-09-16T02:15:38.986081+0000","last_clean":"2024-09-16T02:15:38.986081+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:15:38.986081+0000","last_undegraded":"2024-09-16T02:15:38.986081+0000","last_fullsized":"2024-09-16T02:15:38.986081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'29","last_scrub_stamp":"2024-09-16T02:15:27.850068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.850068+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:50.020929+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0014552569999999999,"stat_sum":{"num_bytes":3162112,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7055,"num_write":13,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.b","version":"110'82","reported_seq":498,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.012587+0000","last_change":"2024-09-16T02:15:39.014614+0000","last_active":"2024-09-16T02:15:39.012587+0000","last_peered":"2024-09-16T02:15:39.012587+0000","last_clean":"2024-09-16T02:15:39.012587+0000","last_became_active":"2024-09-16T02:15:13.074136+0000","last_became_peered":"2024-09-16T02:15:13.074136+0000","last_unstale":"2024-09-16T02:15:39.012587+0000","last_undegraded":"2024-09-16T02:15:39.012587+0000","last_fullsized":"2024-09-16T02:15:39.012587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'73","last_scrub_stamp":"2024-09-16T02:15:37.080155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.080155+0000","objects_scrubbed":6,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:37.299907+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0.02296316,"stat_sum":{"num_bytes":10289152,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":12154,"num_write":55,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8077312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,2,13],"acting":[7,15,1,9,8,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.a","version":"0'0","reported_seq":386,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.000637+0000","last_change":"2024-09-16T02:15:39.001525+0000","last_active":"2024-09-16T02:15:39.000637+0000","last_peered":"2024-09-16T02:15:39.000637+0000","last_clean":"2024-09-16T02:15:39.000637+0000","last_became_active":"2024-09-16T02:15:13.070217+0000","last_became_peered":"2024-09-16T02:15:13.070217+0000","last_unstale":"2024-09-16T02:15:39.000637+0000","last_undegraded":"2024-09-16T02:15:39.000637+0000","last_fullsized":"2024-09-16T02:15:39.000637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:37.746545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.746545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:46.187884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015715779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,14,1,9,8,15,2],"acting":[6,14,1,9,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.9","version":"109'30","reported_seq":317,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.993222+0000","last_change":"2024-09-16T02:15:38.993755+0000","last_active":"2024-09-16T02:15:38.993222+0000","last_peered":"2024-09-16T02:15:38.993222+0000","last_clean":"2024-09-16T02:15:38.993222+0000","last_became_active":"2024-09-16T02:14:43.948134+0000","last_became_peered":"2024-09-16T02:14:43.948134+0000","last_unstale":"2024-09-16T02:15:38.993222+0000","last_undegraded":"2024-09-16T02:15:38.993222+0000","last_fullsized":"2024-09-16T02:15:38.993222+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'26","last_scrub_stamp":"2024-09-16T02:15:30.066659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:30.066659+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:35.439319+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.001923983,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13,1,0,8,6,12],"acting":[7,13,1,0,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.8","version":"109'123","reported_seq":521,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.987821+0000","last_change":"2024-09-16T02:15:38.992081+0000","last_active":"2024-09-16T02:15:39.987821+0000","last_peered":"2024-09-16T02:15:39.987821+0000","last_clean":"2024-09-16T02:15:39.987821+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:15:39.987821+0000","last_undegraded":"2024-09-16T02:15:39.987821+0000","last_fullsized":"2024-09-16T02:15:39.987821+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:42.809902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001455411,"stat_sum":{"num_bytes":9732096,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":24888,"num_write":67,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"1.0","version":"26'112","reported_seq":317,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.043898+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:15:40.043898+0000","last_peered":"2024-09-16T02:15:40.043898+0000","last_clean":"2024-09-16T02:15:40.043898+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:15:40.043898+0000","last_undegraded":"2024-09-16T02:15:40.043898+0000","last_fullsized":"2024-09-16T02:15:40.043898+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":187,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.988141+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:15:39.988141+0000","last_peered":"2024-09-16T02:15:39.988141+0000","last_clean":"2024-09-16T02:15:39.988141+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:15:39.988141+0000","last_undegraded":"2024-09-16T02:15:39.988141+0000","last_fullsized":"2024-09-16T02:15:39.988141+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'178","reported_seq":621,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.044038+0000","last_change":"2024-09-16T02:15:39.021388+0000","last_active":"2024-09-16T02:15:40.044038+0000","last_peered":"2024-09-16T02:15:40.044038+0000","last_clean":"2024-09-16T02:15:40.044038+0000","last_became_active":"2024-09-16T02:15:13.072281+0000","last_became_peered":"2024-09-16T02:15:13.072281+0000","last_unstale":"2024-09-16T02:15:40.044038+0000","last_undegraded":"2024-09-16T02:15:40.044038+0000","last_fullsized":"2024-09-16T02:15:40.044038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'161","last_scrub_stamp":"2024-09-16T02:15:37.976949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.976949+0000","objects_scrubbed":14,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:01.555650+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0.026017814,"stat_sum":{"num_bytes":18137088,"num_objects":16,"num_object_clones":9,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":72,"num_read_kb":17166,"num_write":134,"num_write_kb":21728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,4,14,1,0],"acting":[13,12,6,4,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":184,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.987302+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:15:39.987302+0000","last_peered":"2024-09-16T02:15:39.987302+0000","last_clean":"2024-09-16T02:15:39.987302+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:15:39.987302+0000","last_undegraded":"2024-09-16T02:15:39.987302+0000","last_fullsized":"2024-09-16T02:15:39.987302+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:25.996964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'241","reported_seq":666,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.047089+0000","last_change":"2024-09-16T02:15:39.022506+0000","last_active":"2024-09-16T02:15:40.047089+0000","last_peered":"2024-09-16T02:15:40.047089+0000","last_clean":"2024-09-16T02:15:40.047089+0000","last_became_active":"2024-09-16T02:14:58.940506+0000","last_became_peered":"2024-09-16T02:14:58.940506+0000","last_unstale":"2024-09-16T02:15:40.047089+0000","last_undegraded":"2024-09-16T02:15:40.047089+0000","last_fullsized":"2024-09-16T02:15:40.047089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'209","last_scrub_stamp":"2024-09-16T02:15:29.441137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:29.441137+0000","objects_scrubbed":13,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:39.286585+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.022827719,"stat_sum":{"num_bytes":18284544,"num_objects":13,"num_object_clones":7,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":55,"num_read_kb":17943,"num_write":190,"num_write_kb":25232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":181,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.044097+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:15:40.044097+0000","last_peered":"2024-09-16T02:15:40.044097+0000","last_clean":"2024-09-16T02:15:40.044097+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:15:40.044097+0000","last_undegraded":"2024-09-16T02:15:40.044097+0000","last_fullsized":"2024-09-16T02:15:40.044097+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:20.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"48'8","reported_seq":381,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.047791+0000","last_change":"2024-09-16T02:15:39.006681+0000","last_active":"2024-09-16T02:15:40.047791+0000","last_peered":"2024-09-16T02:15:40.047791+0000","last_clean":"2024-09-16T02:15:40.047791+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:15:40.047791+0000","last_undegraded":"2024-09-16T02:15:40.047791+0000","last_fullsized":"2024-09-16T02:15:40.047791+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:41.574147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082058989999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.2","version":"28'2","reported_seq":194,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.987338+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:15:39.987338+0000","last_peered":"2024-09-16T02:15:39.987338+0000","last_clean":"2024-09-16T02:15:39.987338+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:15:39.987338+0000","last_undegraded":"2024-09-16T02:15:39.987338+0000","last_fullsized":"2024-09-16T02:15:39.987338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"109'31","reported_seq":478,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.057360+0000","last_change":"2024-09-16T02:15:40.057360+0000","last_active":"2024-09-16T02:15:40.057360+0000","last_peered":"2024-09-16T02:15:40.057360+0000","last_clean":"2024-09-16T02:15:40.057360+0000","last_became_active":"2024-09-16T02:15:13.074572+0000","last_became_peered":"2024-09-16T02:15:13.074572+0000","last_unstale":"2024-09-16T02:15:40.057360+0000","last_undegraded":"2024-09-16T02:15:40.057360+0000","last_fullsized":"2024-09-16T02:15:40.057360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'31","last_scrub_stamp":"2024-09-16T02:15:40.057220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:40.057220+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:54.439810+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0015176969999999999,"stat_sum":{"num_bytes":1687552,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":5328,"num_write":29,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,12],"acting":[13,2,11,14,4,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.5","version":"110'81","reported_seq":201,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.047846+0000","last_change":"2024-09-16T02:15:39.053755+0000","last_active":"2024-09-16T02:15:40.047846+0000","last_peered":"2024-09-16T02:15:40.047846+0000","last_clean":"2024-09-16T02:15:40.047846+0000","last_became_active":"2024-09-16T02:15:13.073157+0000","last_became_peered":"2024-09-16T02:15:13.073157+0000","last_unstale":"2024-09-16T02:15:40.047846+0000","last_undegraded":"2024-09-16T02:15:40.047846+0000","last_fullsized":"2024-09-16T02:15:40.047846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:41.508130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.055262523000000001,"stat_sum":{"num_bytes":9093120,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":46,"num_read_kb":33437,"num_write":39,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5144576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,15,14,0,7,8],"acting":[12,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":98,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:40.047122+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:15:40.047122+0000","last_peered":"2024-09-16T02:15:40.047122+0000","last_clean":"2024-09-16T02:15:40.047122+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:15:40.047122+0000","last_undegraded":"2024-09-16T02:15:40.047122+0000","last_fullsized":"2024-09-16T02:15:40.047122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"110'106","reported_seq":522,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.944949+0000","last_change":"2024-09-16T02:15:39.944949+0000","last_active":"2024-09-16T02:15:39.944949+0000","last_peered":"2024-09-16T02:15:39.944949+0000","last_clean":"2024-09-16T02:15:39.944949+0000","last_became_active":"2024-09-16T02:14:55.933032+0000","last_became_peered":"2024-09-16T02:14:55.933032+0000","last_unstale":"2024-09-16T02:15:39.944949+0000","last_undegraded":"2024-09-16T02:15:39.944949+0000","last_fullsized":"2024-09-16T02:15:39.944949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'106","last_scrub_stamp":"2024-09-16T02:15:39.944501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:39.944501+0000","objects_scrubbed":7,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:07.530832+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.020947415,"stat_sum":{"num_bytes":4014080,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":1359,"num_write":93,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4112384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":182,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.989102+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:15:38.989102+0000","last_peered":"2024-09-16T02:15:38.989102+0000","last_clean":"2024-09-16T02:15:38.989102+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:15:38.989102+0000","last_undegraded":"2024-09-16T02:15:38.989102+0000","last_fullsized":"2024-09-16T02:15:38.989102+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:21.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"110'152","reported_seq":522,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.988205+0000","last_change":"2024-09-16T02:15:39.398146+0000","last_active":"2024-09-16T02:15:39.988205+0000","last_peered":"2024-09-16T02:15:39.988205+0000","last_clean":"2024-09-16T02:15:39.988205+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:15:39.988205+0000","last_undegraded":"2024-09-16T02:15:39.988205+0000","last_fullsized":"2024-09-16T02:15:39.988205+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'144","last_scrub_stamp":"2024-09-16T02:15:38.304158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:38.304158+0000","objects_scrubbed":10,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:04.664921+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0.41430857300000001,"stat_sum":{"num_bytes":9994240,"num_objects":10,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":26,"num_read_kb":2074,"num_write":84,"num_write_kb":13776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":191,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.994121+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:15:38.994121+0000","last_peered":"2024-09-16T02:15:38.994121+0000","last_clean":"2024-09-16T02:15:38.994121+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:15:38.994121+0000","last_undegraded":"2024-09-16T02:15:38.994121+0000","last_fullsized":"2024-09-16T02:15:38.994121+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"110'49","reported_seq":434,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:39.004424+0000","last_change":"2024-09-16T02:15:39.006345+0000","last_active":"2024-09-16T02:15:39.004424+0000","last_peered":"2024-09-16T02:15:39.004424+0000","last_clean":"2024-09-16T02:15:39.004424+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:15:39.004424+0000","last_undegraded":"2024-09-16T02:15:39.004424+0000","last_fullsized":"2024-09-16T02:15:39.004424+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'45","last_scrub_stamp":"2024-09-16T02:15:36.983585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:36.983585+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:53.480085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.012849442000000001,"stat_sum":{"num_bytes":2506752,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":1828,"num_write":36,"num_write_kb":2736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":190,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-16T02:15:38.999925+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:15:38.999925+0000","last_peered":"2024-09-16T02:15:38.999925+0000","last_clean":"2024-09-16T02:15:38.999925+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:15:38.999925+0000","last_undegraded":"2024-09-16T02:15:38.999925+0000","last_fullsized":"2024-09-16T02:15:38.999925+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113639424,"num_objects":95,"num_object_clones":48,"num_object_copies":665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":6,"num_read":452,"num_read_kb":159719,"num_write":933,"num_write_kb":153504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202911744,"data_stored":279519232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},"log_size":1384,"ondisk_log_size":1384,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280534,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47684,"kb_used_data":12336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324156,"statfs":{"total":96636764160,"available":96587935744,"internally_reserved":0,"allocated":12632064,"data_stored":19312195,"data_compressed":294632,"data_compressed_allocated":401408,"data_compressed_original":1232896,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280534,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44676,"kb_used_data":13484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327164,"statfs":{"total":96636764160,"available":96591015936,"internally_reserved":0,"allocated":13807616,"data_stored":19737635,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280535,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45296,"kb_used_data":9944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326544,"statfs":{"total":96636764160,"available":96590381056,"internally_reserved":0,"allocated":10182656,"data_stored":13417507,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313239,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":49920,"kb_used_data":18732,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321920,"statfs":{"total":96636764160,"available":96585646080,"internally_reserved":0,"allocated":19181568,"data_stored":23839561,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280535,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47108,"kb_used_data":15916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324732,"statfs":{"total":96636764160,"available":96588525568,"internally_reserved":0,"allocated":16297984,"data_stored":22510627,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313237,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48472,"kb_used_data":17284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323368,"statfs":{"total":96636764160,"available":96587128832,"internally_reserved":0,"allocated":17698816,"data_stored":22803273,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280534,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41676,"kb_used_data":10488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330164,"statfs":{"total":96636764160,"available":96594087936,"internally_reserved":0,"allocated":10739712,"data_stored":15614841,"data_compressed":265137,"data_compressed_allocated":364544,"data_compressed_original":729088,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280534,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42136,"kb_used_data":10944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329704,"statfs":{"total":96636764160,"available":96593616896,"internally_reserved":0,"allocated":11206656,"data_stored":13253667,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280533,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46996,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324844,"statfs":{"total":96636764160,"available":96588640256,"internally_reserved":0,"allocated":16183296,"data_stored":22088758,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280534,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39760,"kb_used_data":12736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94332080,"statfs":{"total":96636764160,"available":96596049920,"internally_reserved":0,"allocated":13041664,"data_stored":16335689,"data_compressed":259063,"data_compressed_allocated":356352,"data_compressed_original":712704,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280534,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51084,"kb_used_data":19896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320756,"statfs":{"total":96636764160,"available":96584454144,"internally_reserved":0,"allocated":20373504,"data_stored":26795587,"data_compressed":294632,"data_compressed_allocated":401408,"data_compressed_original":1232896,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44840,"kb_used_data":13648,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327000,"statfs":{"total":96636764160,"available":96590848000,"internally_reserved":0,"allocated":13975552,"data_stored":18127907,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280534,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37428,"kb_used_data":10400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334412,"statfs":{"total":96636764160,"available":96598437888,"internally_reserved":0,"allocated":10649600,"data_stored":13261859,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280534,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46140,"kb_used_data":14948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325700,"statfs":{"total":96636764160,"available":96589516800,"internally_reserved":0,"allocated":15306752,"data_stored":20745251,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280534,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45304,"kb_used_data":14116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326536,"statfs":{"total":96636764160,"available":96590372864,"internally_reserved":0,"allocated":14454784,"data_stored":18725923,"data_compressed":262101,"data_compressed_allocated":360448,"data_compressed_original":720896,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":0,"seq":429496729604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18853888,"data_stored":25047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15015936,"data_stored":21540864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":14635008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12517376,"data_stored":16908288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9367552,"data_stored":12292096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9924608,"data_stored":12283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13848576,"data_stored":20054016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14897152,"data_stored":21118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11816960,"data_stored":16326656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13172736,"data_stored":17756160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7139328,"data_stored":8015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8900608,"data_stored":12447744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":18518016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11288576,"data_stored":17813504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":17948672,"data_stored":22917120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":16429056,"data_stored":21843968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:15:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:44.350+0000 7ff756ffd640 0 -- 172.21.15.3:0/3453696589 >> [v2:172.21.15.177:6800/1752008388,v1:172.21.15.177:6801/1752008388] conn(0x7ff7582222a0 msgr2=0x7ff758224690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:44.372 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:44.372 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:44.387 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:44.438 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:44.438+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:44.505 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:44.505 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:44.519 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:44.539 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:44.538+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:44.640 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:44.638+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:44.643 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:44.643 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:44.657 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:44.740 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:44.737+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:44.781 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:44.781 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:44.795 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:44.841 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:44.840+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:44.942 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:44.936+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:44.961 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:44.961 INFO:teuthology.orchestra.run.smithi195.stdout: "num_ops": 0 2024-09-16T02:15:44.961 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:44.975 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:45.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-16T02:15:45.042 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:45.037+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:45.100 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:45.100 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [], 2024-09-16T02:15:45.100 INFO:teuthology.orchestra.run.smithi195.stdout: "complaint_time": 30, 2024-09-16T02:15:45.100 INFO:teuthology.orchestra.run.smithi195.stdout: "num_blocked_ops": 0 2024-09-16T02:15:45.100 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:45.114 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:45.143 INFO:tasks.ceph.osd.10.smithi177.stderr:2024-09-16T02:15:45.141+0000 7f2aed387640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 32947) UID: 0 2024-09-16T02:15:45.243 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:45.237+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:45.281 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-09-16T02:15:45.281 INFO:teuthology.orchestra.run.smithi195.stdout: "size": 20, 2024-09-16T02:15:45.281 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 600, 2024-09-16T02:15:45.281 INFO:teuthology.orchestra.run.smithi195.stdout: "ops": [ 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:16 3.6s0 3:6fc76ee5:::smithi19530025-6:head [append 0~98304,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.998528299, 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017465325, 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 16 2024-09-16T02:15:45.282 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279235+0000", 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.283 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279238+0000", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.8100000000000002e-06 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279326+0000", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.8633000000000003e-05 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279349+0000", 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2527e-05 2024-09-16T02:15:45.284 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279366+0000", 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7144999999999999e-05 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279719+0000", 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00035280199999999999 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.285 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279922+0000", 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020340399999999999 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.281506+0000", 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015834829999999999 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285302+0000", 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0037957339999999998 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.286 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296623+0000", 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011321197 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296700+0000", 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7589999999999994e-05 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.287 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:17 3.6s0 3:6fc76ee5:::smithi19530025-6:head [setxattr _header (55) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.998335523999998, 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017531332, 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.288 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 17 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279428+0000", 2024-09-16T02:15:45.289 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279429+0000", 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.268e-06 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279437+0000", 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990000000000003e-06 2024-09-16T02:15:45.290 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279454+0000", 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7649999999999999e-05 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279467+0000", 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2541000000000001e-05 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.291 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.283947+0000", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0044801099999999998 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284098+0000", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015146199999999999 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.284983+0000", 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00088467199999999998 2024-09-16T02:15:45.292 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285989+0000", 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010060710000000001 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296878+0000", 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.010888273 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.293 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.296959+0000", 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1586000000000001e-05 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:18 3.6s0 3:6fc76ee5:::smithi19530025-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.998277481000002, 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.024399387000000002, 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.294 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 18 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.295 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279486+0000", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279487+0000", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7000000000000003e-07 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279491+0000", 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.1640000000000003e-06 2024-09-16T02:15:45.296 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279517+0000", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.6256999999999999e-05 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.279528+0000", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1083999999999999e-05 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285819+0000", 2024-09-16T02:15:45.297 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0062906460000000004 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.285890+0000", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1496999999999997e-05 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297729+0000", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.011838918 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.297881+0000", 2024-09-16T02:15:45.298 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00015223 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.303885+0000", 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0060037210000000001 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:53 3.6s0 3:63499ffe:::smithi19530025-18:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.969120551000003, 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.062535054000000007, 2024-09-16T02:15:45.299 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 53 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.300 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308643+0000", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.5300000000000004e-07 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308647+0000", 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.3359999999999999e-06 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.301 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308655+0000", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5269999999999998e-06 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308660+0000", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2170000000000002e-06 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311674+0000", 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0030135090000000002 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.302 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.311727+0000", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2865000000000003e-05 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312012+0000", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00028502200000000001 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312750+0000", 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00073797000000000001 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.303 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371099+0000", 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.05834901 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371178+0000", 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8944999999999997e-05 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.304 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:54 3.6s0 3:63499ffe:::smithi19530025-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.969067491999994, 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.063438222000000002, 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 54 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.305 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308696+0000", 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.51e-07 2024-09-16T02:15:45.306 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308699+0000", 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.711e-06 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308707+0000", 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.9209999999999992e-06 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.307 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.308720+0000", 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2841e-05 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312277+0000", 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0035568430000000001 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.312325+0000", 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8183999999999998e-05 2024-09-16T02:15:45.308 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371447+0000", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.059122365000000003 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.371532+0000", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.5254999999999996e-05 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.372134+0000", 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00060155099999999997 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.309 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.5s2 35/32 ECSubWrite(tid=10, reqid=client.4598.0:74, at_version=35'8, trim_to=0'0, roll_forward_to=35'4))", 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 80.933886029000007, 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.144956114, 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.310 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343877+0000", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343878+0000", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2799999999999996e-07 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.311 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.343889+0000", 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1214e-05 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488006+0000", 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.14411742599999999 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488019+0000", 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2075e-05 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.312 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488095+0000", 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.6990999999999995e-05 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488109+0000", 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3475000000000001e-05 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.313 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488796+0000", 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00068681199999999997 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:24.488833+0000", 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7592999999999997e-05 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:262 3.6s0 3:63499ffe:::smithi19530025-18:head [append 0~0] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:45.314 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 69.920673331000003, 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016886259000000001, 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 262 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:45.315 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357090+0000", 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357093+0000", 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1700000000000001e-06 2024-09-16T02:15:45.316 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357107+0000", 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4207e-05 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357142+0000", 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.4171999999999997e-05 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.317 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357158+0000", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6135999999999999e-05 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357305+0000", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000147169 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "op must wait for map", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357326+0000", 2024-09-16T02:15:45.318 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1352000000000002e-05 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.361983+0000", 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0046572769999999996 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.362251+0000", 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00026744600000000001 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.319 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.364433+0000", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0021825460000000001 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370083+0000", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0056496740000000004 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373776+0000", 2024-09-16T02:15:45.320 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0036930560000000001 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.373976+0000", 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020005399999999999 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.321 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:263 3.6s0 3:63499ffe:::smithi19530025-18:head [rollback 7] snapc 7=[7] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 69.920564093999999, 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017578257, 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.322 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 263 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357199+0000", 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.323 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357201+0000", 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.8899999999999999e-06 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357225+0000", 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3842e-05 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.324 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357247+0000", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2096e-05 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357262+0000", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5277e-05 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357490+0000", 2024-09-16T02:15:45.325 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000227483 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.357502+0000", 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.1773000000000001e-05 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365565+0000", 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0080633289999999993 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.326 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.365942+0000", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00037663200000000001 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.367097+0000", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.001155468 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.370086+0000", 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0029886019999999999 2024-09-16T02:15:45.327 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374668+0000", 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.004582315 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:35.374777+0000", 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010955000000000001 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.328 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:397 3.6s0 3:63499ffe:::smithi19530025-18:head [copy-from ver 4 in=55b] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 64.371299913000001, 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79546935399999996, 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.329 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 397 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906463+0000", 2024-09-16T02:15:45.330 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906466+0000", 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7839999999999999e-06 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906481+0000", 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4399000000000001e-05 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.331 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906518+0000", 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.7645999999999998e-05 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906539+0000", 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1189e-05 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906622+0000", 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.2161000000000002e-05 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.332 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906742+0000", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00012039100000000001 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.925051+0000", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018308787999999999 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.929002+0000", 2024-09-16T02:15:45.333 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0039508540000000002 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701787+0000", 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77278573299999997 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701933+0000", 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014540899999999999 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.334 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:398 3.6s0 3:63499ffe:::smithi19530025-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 64.371184185999994, 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.021958095, 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "started", 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 398 2024-09-16T02:15:45.335 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906579+0000", 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.336 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906580+0000", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.204e-06 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906589+0000", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.6659999999999997e-06 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906610+0000", 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1492999999999999e-05 2024-09-16T02:15:45.337 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906624+0000", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.3923e-05 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.906937+0000", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00031242800000000001 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.907028+0000", 2024-09-16T02:15:45.338 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.1557000000000004e-05 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.926821+0000", 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.019792588 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.927569+0000", 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00074833600000000005 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.339 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928537+0000", 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00096790000000000005 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 52/32 ECSubWriteReply(tid=25, last_complete=52'15, committed=1, applied=1))", 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 64.349649403000001, 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77450966499999996, 2024-09-16T02:15:45.340 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928114+0000", 2024-09-16T02:15:45.341 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928121+0000", 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.4100000000000002e-06 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:40.928131+0000", 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 9.4860000000000003e-06 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.342 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701464+0000", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.77333287100000003 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701480+0000", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6025999999999999e-05 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.701649+0000", 2024-09-16T02:15:45.343 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016903900000000001 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:14:41.702624+0000", 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00097483300000000004 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.es6 73/66 ECSubWrite(tid=66, reqid=osd.8.0:66, at_version=73'42, trim_to=0'0, roll_forward_to=72'38))", 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:45.344 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 43.322817358000002, 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.018190284000000001, 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.345 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954946+0000", 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954947+0000", 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5069999999999999e-06 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.346 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.954992+0000", 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.4162000000000002e-05 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955233+0000", 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000241566 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.955251+0000", 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7448999999999998e-05 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.347 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956483+0000", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0012321509999999999 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.956507+0000", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4029999999999999e-05 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973035+0000", 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016528547000000001 2024-09-16T02:15:45.348 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.973136+0000", 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00010087200000000001 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.8s4 73/32 ECSubWrite(tid=107, reqid=osd.5.0:107, at_version=73'75, trim_to=0'0, roll_forward_to=73'73))", 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 43.319108817, 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016895095999999998, 2024-09-16T02:15:45.349 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958654+0000", 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.350 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958656+0000", 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5939999999999999e-06 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958707+0000", 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.1008999999999999e-05 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.351 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958930+0000", 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00022282899999999999 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.958952+0000", 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2146000000000001e-05 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959126+0000", 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000173807 2024-09-16T02:15:45.352 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.959156+0000", 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.0022000000000002e-05 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975497+0000", 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.016340924999999999 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.353 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.975550+0000", 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 5.2763999999999998e-05 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 73/66 ECSubWrite(tid=103, reqid=client.4598.0:646, at_version=73'28, trim_to=0'0, roll_forward_to=71'26))", 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 43.313662563000001, 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.014562088000000001, 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.354 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964101+0000", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964103+0000", 2024-09-16T02:15:45.355 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.26e-06 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964341+0000", 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00023783300000000001 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964452+0000", 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000110974 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.356 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964476+0000", 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.393e-05 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964624+0000", 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00014846400000000001 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.964652+0000", 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.7389000000000002e-05 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.357 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978619+0000", 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.013967228 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:01.978663+0000", 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.401e-05 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.358 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.4s5 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 37.211932533000002, 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79668697799999999, 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.359 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065831+0000", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065832+0000", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.4780000000000001e-06 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.065862+0000", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.9570999999999999e-05 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.360 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860567+0000", 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.79470507099999999 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860588+0000", 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.0665e-05 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860756+0000", 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016797700000000001 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.361 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.860787+0000", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 3.1492000000000003e-05 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862431+0000", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0016435880000000001 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:08.862518+0000", 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.7136000000000001e-05 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.362 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=103, reqid=client.4598.0:770, at_version=87'86, trim_to=0'0, roll_forward_to=79'84))", 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 29.319059607, 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.017445961999999999, 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.363 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958704+0000", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.958706+0000", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.1660000000000001e-06 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.959284+0000", 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00057834499999999997 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.364 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972561+0000", 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.01327654 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972585+0000", 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.4221999999999999e-05 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.365 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972741+0000", 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000156248 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.972764+0000", 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.3136e-05 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976076+0000", 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.003311858 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.366 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.976150+0000", 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.3447000000000003e-05 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWrite(3.ds4 87/32 ECSubWrite(tid=107, reqid=client.4598.0:774, at_version=87'90, trim_to=0'0, roll_forward_to=87'86))", 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 29.292561053, 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.19572739, 2024-09-16T02:15:45.367 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985202+0000", 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.368 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985204+0000", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 2.2630000000000002e-06 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:15.985253+0000", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.8288000000000003e-05 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179555+0000", 2024-09-16T02:15:45.369 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.194302157 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179570+0000", 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.5322999999999999e-05 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179746+0000", 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000175391 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.370 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.179762+0000", 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6388999999999998e-05 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180848+0000", 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0010859349999999999 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:16.180930+0000", 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.1644000000000006e-05 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.371 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "osd_op(client.4598.0:831 3.6s0 3:63499ffe:::smithi19530025-18:head [delete] snapc 1d=[1d,1c,18,d] ondisk+write+known_if_redirected+full_force+supports_pool_eio e91)", 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 25.271924776999999, 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.89186369399999998, 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_info": { 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client": "client.4598", 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "tid": 831 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.372 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.373 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005839+0000", 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 8.0999999999999997e-07 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005846+0000", 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.1690000000000003e-06 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005911+0000", 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 6.4146000000000006e-05 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.374 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.005928+0000", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.7541999999999999e-05 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008320+0000", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.002391476 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "started", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008367+0000", 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 4.757e-05 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.375 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_started", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.008731+0000", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00036390000000000001 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "sub_op_committed", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.009626+0000", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00089470299999999995 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "commit_sent", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897500+0000", 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88787382500000001 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.376 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897702+0000", 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00020255300000000001 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 25.267712692, 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066496243999999996, 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.377 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010051+0000", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010052+0000", 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.0899999999999999e-06 2024-09-16T02:15:45.378 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010064+0000", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.2726e-05 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076273+0000", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.066208820000000002 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076290+0000", 2024-09-16T02:15:45.379 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.6654e-05 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076469+0000", 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.000178717 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.076547+0000", 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.8237000000000003e-05 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.380 INFO:teuthology.orchestra.run.smithi195.stdout: "description": "MOSDECSubOpWriteReply(3.6s0 91/32 ECSubWriteReply(tid=51, last_complete=91'33, committed=1, applied=1))", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "initiated_at": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "age": 25.266833034000001, 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88785698300000004, 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "continuous": false, 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "type_data": { 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "flag_point": "reached pg", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "events": [ 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "initiated", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "header_read", 2024-09-16T02:15:45.381 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010930+0000", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "throttled", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010932+0000", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.48e-06 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "all_read", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.010939+0000", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 7.7160000000000003e-06 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "dispatched", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897090+0000", 2024-09-16T02:15:45.382 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.88615089199999997 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "queued_for_pg", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897108+0000", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 1.77e-05 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "reached_pg", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.897272+0000", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.00016364099999999999 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: }, 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: { 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "event": "done", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "time": "2024-09-16T02:15:20.898787+0000", 2024-09-16T02:15:45.383 INFO:teuthology.orchestra.run.smithi195.stdout: "duration": 0.0015155539999999999 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout: } 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-09-16T02:15:45.384 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-09-16T02:15:45.385 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:45.386 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:45.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:45.387 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:45.388 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:45.388 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:45.380+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:45.486 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:45.486+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:45.523 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:45.523 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:45.537 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:15:45.588 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:45.584+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:45.659 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:45.659 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:45.673 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:15:45.717 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:45.717+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:45.789 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:45.785+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:45.796 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:45.797 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:45.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-16T02:15:45.891 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:45.889+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:45.920 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-09-16T02:15:45.944 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:45.944 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:45.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-16T02:15:45.998 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:45.998+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:46.093 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:46.090+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:46.120 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.120 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-16T02:15:46.194 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:46.189+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:46.295 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:46.292+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:46.304 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.304 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-16T02:15:46.395 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:46.393+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:46.444 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.444 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-16T02:15:46.496 INFO:tasks.ceph.osd.15.smithi195.stderr:2024-09-16T02:15:46.493+0000 7f1dc3b6a640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26583) UID: 0 2024-09-16T02:15:46.583 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.583 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.596 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:46.594+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:46.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-16T02:15:46.697 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:46.697+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:46.757 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.757 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-16T02:15:46.798 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:46.796+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:46.891 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:46.891 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:46.899 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:46.896+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:46.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-16T02:15:47.000 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:46.998+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:47.035 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:47.035 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:47.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-16T02:15:47.101 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:47.096+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:47.202 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:47.196+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:47.225 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:47.225 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:47.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-16T02:15:47.303 INFO:tasks.ceph.osd.8.smithi003.stderr:2024-09-16T02:15:47.302+0000 7f96b1b9d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26329) UID: 0 2024-09-16T02:15:47.361 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:47.361 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:47.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-16T02:15:47.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-16T02:15:47.403 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:47.397+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:47.504 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:47.501+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:47.538 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:47.538 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:47.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-16T02:15:47.580 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:47.580 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:47.605 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:47.601+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:47.606 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:47.606 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:47.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:47.632 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:47.659 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:47.659 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:47.670 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:47.670 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:47.684 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:47.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-16T02:15:47.706 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:47.705+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:47.712 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:47.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:47.741 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:47.741 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:47.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:47.773 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:47.802 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:47.802 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:47.807 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:47.806+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:47.816 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:47.817 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:47.817 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:47.817 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:47.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-16T02:15:47.908 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:47.904+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:47.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:15:47.952 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:47.952 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:47.953 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:47.953 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:47.953 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:47.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-16T02:15:48.009 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:48.005+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:48.037 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:48.081 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:48.081 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:48.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:48.109 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:48.110 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:48.104+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:48.129 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:48.129 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:48.139 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=22, reqid=client.4598.0:404, at_version=52'7, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 67.168890605000001, 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013916968, 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.140 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967630+0000", 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967631+0000", 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.424e-06 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.967931+0000", 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030039800000000001 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979030+0000", 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011099121999999999 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979051+0000", 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0577e-05 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979192+0000", 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140526 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.979207+0000", 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5801999999999999e-05 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981459+0000", 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022516110000000002 2024-09-16T02:15:48.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.981546+0000", 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7508000000000003e-05 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.9s1 52/32 ECSubWrite(tid=25, reqid=client.4598.0:407, at_version=52'10, trim_to=0'0, roll_forward_to=45'6))", 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:48.145 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 67.152307618999998, 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34897630299999999, 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984213+0000", 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984214+0000", 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.384e-06 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:40.984244+0000", 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0549000000000002e-05 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331741+0000", 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.34749696499999999 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331756+0000", 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4873e-05 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331921+0000", 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016467900000000001 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.331952+0000", 2024-09-16T02:15:48.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0895000000000002e-05 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333108+0000", 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001156633 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:41.333189+0000", 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0325000000000001e-05 2024-09-16T02:15:48.150 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=109, reqid=osd.5.0:109, at_version=74'77, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.180768317999998, 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060351959999999996, 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.151 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955752+0000", 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3600000000000003e-07 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955857+0000", 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010480999999999999 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.955998+0000", 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014078300000000001 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956026+0000", 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8285000000000001e-05 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956200+0000", 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000174165 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.956232+0000", 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1047000000000003e-05 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961698+0000", 2024-09-16T02:15:48.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054662360000000002 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.961787+0000", 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9234000000000007e-05 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=110, reqid=osd.5.0:110, at_version=74'79, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:48.156 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.176895283999997, 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060372860000000002, 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959625+0000", 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959627+0000", 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9099999999999999e-06 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959685+0000", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8099000000000001e-05 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959791+0000", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106409 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959814+0000", 2024-09-16T02:15:48.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3224999999999999e-05 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960393+0000", 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057815700000000004 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960417+0000", 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4808e-05 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965585+0000", 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051671030000000001 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965662+0000", 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7575000000000002e-05 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 74/32 ECSubWrite(tid=277, reqid=osd.13.0:277, at_version=74'88, trim_to=0'0, roll_forward_to=73'84))", 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.175861226999999, 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057458700000000001, 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960659+0000", 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960660+0000", 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0400000000000005e-07 2024-09-16T02:15:48.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960691+0000", 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1438e-05 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960918+0000", 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227304 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960949+0000", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0141000000000002e-05 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962083+0000", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011340129999999999 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962117+0000", 2024-09-16T02:15:48.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3921999999999998e-05 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966316+0000", 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041993250000000003 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.966405+0000", 2024-09-16T02:15:48.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8923000000000002e-05 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 74/32 ECSubWrite(tid=114, reqid=osd.7.0:114, at_version=74'29, trim_to=0'0, roll_forward_to=73'27))", 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.171001298, 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085142380000000004, 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.169 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965519+0000", 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965520+0000", 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4e-07 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965569+0000", 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9175999999999998e-05 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965669+0000", 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9946999999999997e-05 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965692+0000", 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3422e-05 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965858+0000", 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016617800000000001 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965887+0000", 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8781000000000001e-05 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973464+0000", 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075773079999999996 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974033+0000", 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00056858600000000005 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 74/32 ECSubWrite(tid=112, reqid=client.4598.0:655, at_version=74'82, trim_to=0'0, roll_forward_to=73'75))", 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.167842010000001, 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062268389999999996, 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:48.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968678+0000", 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968680+0000", 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.903e-06 2024-09-16T02:15:48.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968727+0000", 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7139999999999999e-05 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968819+0000", 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1711000000000003e-05 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968841+0000", 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2314e-05 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969011+0000", 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000169872 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.969038+0000", 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6755e-05 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974822+0000", 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005783697 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974905+0000", 2024-09-16T02:15:48.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3447000000000003e-05 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.es2 74/66 ECSubWrite(tid=68, reqid=client.4598.0:653, at_version=74'44, trim_to=0'0, roll_forward_to=73'42))", 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.163401411999999, 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069659270000000002, 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.180 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973119+0000", 2024-09-16T02:15:48.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973120+0000", 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7399999999999991e-07 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973262+0000", 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000142483 2024-09-16T02:15:48.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973686+0000", 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042359299999999999 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973721+0000", 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5491000000000003e-05 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.973886+0000", 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000164372 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974322+0000", 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000436556 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.979991+0000", 2024-09-16T02:15:48.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056687619999999999 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980085+0000", 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3696000000000006e-05 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 79/32 ECSubWrite(tid=138, reqid=osd.7.0:138, at_version=79'38, trim_to=0'0, roll_forward_to=78'36))", 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:48.185 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 40.146264997999999, 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079362210000000002, 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990255+0000", 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990256+0000", 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990299+0000", 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2870999999999997e-05 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990371+0000", 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2208999999999999e-05 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990390+0000", 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8802000000000001e-05 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990544+0000", 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000154661 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.990576+0000", 2024-09-16T02:15:48.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1655000000000001e-05 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998130+0000", 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075539659999999996 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998191+0000", 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1407000000000003e-05 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.190 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 40.144826657999999, 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068584240000000001, 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.191 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991693+0000", 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991694+0000", 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7700000000000003e-07 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991735+0000", 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1068000000000001e-05 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991880+0000", 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014438400000000001 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.991902+0000", 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2616000000000002e-05 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992060+0000", 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015784600000000001 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.992086+0000", 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6239000000000001e-05 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998446+0000", 2024-09-16T02:15:48.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063592730000000004 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:07.998552+0000", 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106121 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.196 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 90/66 ECSubWrite(tid=179, reqid=client.4598.0:789, at_version=90'63, trim_to=0'0, roll_forward_to=85'61))", 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.130763095999999, 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077930029999999997, 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:48.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005757+0000", 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005758+0000", 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:15:48.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005822+0000", 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4696999999999994e-05 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005935+0000", 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011267799999999999 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005959+0000", 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4111999999999999e-05 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006140+0000", 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018136899999999999 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006171+0000", 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0889000000000002e-05 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.013473+0000", 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073015670000000001 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.013550+0000", 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7096999999999997e-05 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 91/66 ECSubWrite(tid=187, reqid=osd.7.0:187, at_version=91'71, trim_to=0'0, roll_forward_to=90'69))", 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.186581988, 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070490179999999998, 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.202 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949938+0000", 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.949939+0000", 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.23e-07 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950059+0000", 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119615 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950166+0000", 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000107125 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950187+0000", 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1685999999999999e-05 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950364+0000", 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017645900000000001 2024-09-16T02:15:48.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.950387+0000", 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3547999999999999e-05 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956892+0000", 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065050230000000004 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.956987+0000", 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4739000000000002e-05 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 91/66 ECSubWrite(tid=188, reqid=osd.7.0:188, at_version=91'73, trim_to=0'0, roll_forward_to=90'69))", 2024-09-16T02:15:48.207 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.185186982000001, 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074890160000000002, 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:48.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951333+0000", 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951335+0000", 2024-09-16T02:15:48.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.623e-06 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951384+0000", 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9150999999999997e-05 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951468+0000", 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4196999999999994e-05 2024-09-16T02:15:48.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951490+0000", 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2166000000000001e-05 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951850+0000", 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035987699999999998 2024-09-16T02:15:48.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.951873+0000", 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3329e-05 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.958741+0000", 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068677360000000002 2024-09-16T02:15:48.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.958822+0000", 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0937000000000001e-05 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 102/83 ECSubWrite(tid=220, reqid=client.4598.0:914, at_version=102'60, trim_to=0'0, roll_forward_to=99'58))", 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:30.949133+0000", 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.187387294000001, 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078810319999999996, 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949133+0000", 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949133+0000", 2024-09-16T02:15:48.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949133+0000", 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4999999999999998e-07 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949669+0000", 2024-09-16T02:15:48.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00053535200000000001 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949738+0000", 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9521000000000001e-05 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949758+0000", 2024-09-16T02:15:48.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9493000000000001e-05 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949927+0000", 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016994800000000001 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949956+0000", 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8676000000000001e-05 2024-09-16T02:15:48.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.956780+0000", 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068241329999999996 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.957014+0000", 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023355899999999999 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 107/66 ECSubWrite(tid=247, reqid=osd.7.0:247, at_version=107'100, trim_to=0'0, roll_forward_to=106'98))", 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.929401+0000", 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.207119421, 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092053529999999995, 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929401+0000", 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929401+0000", 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929401+0000", 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9699999999999996e-07 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929453+0000", 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1824999999999999e-05 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930584+0000", 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011314179999999999 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930605+0000", 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1049000000000001e-05 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930750+0000", 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014483000000000001 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930777+0000", 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6188999999999999e-05 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938507+0000", 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077306179999999999 2024-09-16T02:15:48.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938606+0000", 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8926999999999997e-05 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs2 107/83 ECSubWrite(tid=248, reqid=osd.7.0:248, at_version=107'73, trim_to=0'0, roll_forward_to=106'71))", 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.931633+0000", 2024-09-16T02:15:48.225 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.204887144000001, 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010193615, 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931633+0000", 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931633+0000", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931635+0000", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931696+0000", 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1492e-05 2024-09-16T02:15:48.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931788+0000", 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1626999999999995e-05 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931810+0000", 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2399e-05 2024-09-16T02:15:48.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931999+0000", 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018835499999999999 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.932023+0000", 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4085999999999999e-05 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941729+0000", 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097058149999999996 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941827+0000", 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8023999999999995e-05 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.230 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s3 108/66 ECSubWrite(tid=251, reqid=osd.7.0:251, at_version=108'102, trim_to=0'0, roll_forward_to=107'100))", 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:36.930972+0000", 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.20554842, 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064837899999999997, 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.231 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.930972+0000", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.930972+0000", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.930972+0000", 2024-09-16T02:15:48.232 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8299999999999998e-07 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.931013+0000", 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1319000000000002e-05 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.931167+0000", 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000153416 2024-09-16T02:15:48.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.931217+0000", 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0071999999999998e-05 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.931365+0000", 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014836600000000001 2024-09-16T02:15:48.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.931398+0000", 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2682000000000003e-05 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.937346+0000", 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059479449999999996 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.235 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:36.937455+0000", 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109607 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.8s6 109/32 ECSubWrite(tid=162, reqid=osd.5.0:162, at_version=109'123, trim_to=0'0, roll_forward_to=106'121))", 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.943278+0000", 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.193241932999999, 2024-09-16T02:15:48.236 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062949649999999996, 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943278+0000", 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.237 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943278+0000", 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943280+0000", 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943358+0000", 2024-09-16T02:15:48.238 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8036999999999998e-05 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943470+0000", 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112444 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943496+0000", 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5683000000000001e-05 2024-09-16T02:15:48.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943674+0000", 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000177868 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.943696+0000", 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2033000000000001e-05 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.240 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.949502+0000", 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058065549999999997 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.949573+0000", 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0754999999999996e-05 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.241 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 109/83 ECSubWrite(tid=386, reqid=osd.13.0:386, at_version=109'165, trim_to=0'0, roll_forward_to=109'161))", 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.982544+0000", 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.153976018, 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057782550000000002, 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.242 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.982544+0000", 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.982544+0000", 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.982546+0000", 2024-09-16T02:15:48.243 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5340000000000001e-06 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.982620+0000", 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4340999999999999e-05 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.038200+0000", 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055580417 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.244 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.038225+0000", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4439000000000001e-05 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.038405+0000", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018014600000000001 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.038433+0000", 2024-09-16T02:15:48.245 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8008999999999999e-05 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040226+0000", 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001792587 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040327+0000", 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010107699999999999 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.246 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.2s5 109/83 ECSubWrite(tid=388, reqid=client.4598.0:973, at_version=109'167, trim_to=0'0, roll_forward_to=109'163))", 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.040716+0000", 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 10.095803882, 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006799561, 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.247 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040716+0000", 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040716+0000", 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.248 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040718+0000", 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4100000000000001e-06 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040771+0000", 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3182e-05 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.249 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.046096+0000", 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053251169999999999 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.046120+0000", 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4063e-05 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.046271+0000", 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015084899999999999 2024-09-16T02:15:48.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.046290+0000", 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9245999999999998e-05 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.047398+0000", 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011074979999999999 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.047516+0000", 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118196 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.252 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:48.253 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:48.254 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:48.255 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:48.255 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:48.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-16T02:15:48.256 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:48.257 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:48.257 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:48.249+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:48.354 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:48.353+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:48.428 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:48.428 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:48.429 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:48.429 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:48.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-16T02:15:48.455 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:48.453+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:48.556 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:48.553+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:48.558 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:48.559 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:48.559 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:48.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-16T02:15:48.657 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:48.653+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.694 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:504 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~786432] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.694994977999997, 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022327653999999999, 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.695 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 504 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996029+0000", 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.996033+0000", 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1969999999999998e-06 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.997993+0000", 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019604309999999999 2024-09-16T02:15:48.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998029+0000", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5525999999999998e-05 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998044+0000", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5440000000000001e-05 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998168+0000", 2024-09-16T02:15:48.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012395 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:46.998340+0000", 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000171619 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.000528+0000", 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021878459999999998 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004445+0000", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003916966 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018230+0000", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013784892 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.018356+0000", 2024-09-16T02:15:48.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126787 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:505 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~491520] snapc e=[e,d,a] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 61.687653294, 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022250066999999998, 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.701 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 505 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003370+0000", 2024-09-16T02:15:48.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.003374+0000", 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.517e-06 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004387+0000", 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001013445 2024-09-16T02:15:48.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004415+0000", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7115999999999999e-05 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004434+0000", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9966000000000002e-05 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004556+0000", 2024-09-16T02:15:48.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121734 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.004705+0000", 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014862999999999999 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.006184+0000", 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014795570000000001 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.705 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.008703+0000", 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025187090000000001 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025506+0000", 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016802766 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:47.025620+0000", 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114627 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.706 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:587 3.8s0 3:165ce5f6:::smithi19530025-40:head [rollback d] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 52.781744965999998, 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023594436999999999, 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 587 2024-09-16T02:15:48.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909279+0000", 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909280+0000", 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.28e-06 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909287+0000", 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8129999999999998e-06 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909310+0000", 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2731999999999999e-05 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909329+0000", 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9134e-05 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909687+0000", 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035797500000000001 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909698+0000", 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921510+0000", 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011812049999999999 2024-09-16T02:15:48.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.921788+0000", 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027776500000000002 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.712 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.924415+0000", 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00262737 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926686+0000", 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022710270000000001 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.713 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932811+0000", 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061244309999999996 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.932873+0000", 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2630000000000002e-05 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.714 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:588 3.8s0 3:165ce5f6:::smithi19530025-40:head [append 0~0] snapc 12=[12,f,e,d] ondisk+write+known_if_redirected+supports_pool_eio e67)", 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 52.781640621999998, 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.026094544000000001, 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 588 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.715 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909383+0000", 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.716 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909384+0000", 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909391+0000", 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6159999999999998e-06 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909409+0000", 2024-09-16T02:15:48.717 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8335999999999999e-05 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909424+0000", 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.448e-05 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909774+0000", 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035054599999999997 2024-09-16T02:15:48.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.909792+0000", 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.77e-05 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925523+0000", 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015730929000000001 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.719 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.925651+0000", 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012825900000000001 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.926311+0000", 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00065990199999999997 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.927500+0000", 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001188599 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935426+0000", 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079261060000000005 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:55.935478+0000", 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1684000000000002e-05 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.721 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.3s5 72/66 ECSubWrite(tid=268, reqid=client.4598.0:639, at_version=0'0, trim_to=0'0, roll_forward_to=72'21))", 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.731078078000003, 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51318274200000003, 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.722 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959946+0000", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959947+0000", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4300000000000001e-07 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:00.959959+0000", 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2097e-05 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.723 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471943+0000", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.51198411399999999 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.471961+0000", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8641999999999999e-05 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472084+0000", 2024-09-16T02:15:48.724 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012220300000000001 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.472100+0000", 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6583e-05 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473040+0000", 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094022500000000005 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.725 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:01.473128+0000", 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7935000000000004e-05 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:654 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.733564565999998, 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020180719, 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.726 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 654 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.727 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957459+0000", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957462+0000", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7800000000000001e-06 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957473+0000", 2024-09-16T02:15:48.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1266e-05 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957499+0000", 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5978000000000001e-05 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957519+0000", 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545999999999999e-05 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.959927+0000", 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024080640000000001 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.960218+0000", 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029093199999999999 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.962184+0000", 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019666520000000002 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.975499+0000", 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013314994 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977423+0000", 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019231490000000001 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977640+0000", 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021735799999999999 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:655 3.8s0 3:177f2fba:::smithi19530025-45:head [rollback d] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.733477872000002, 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023374418000000001, 2024-09-16T02:15:48.732 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.733 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 655 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:48.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957546+0000", 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957547+0000", 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1399999999999995e-07 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957551+0000", 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1659999999999996e-06 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957562+0000", 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0773e-05 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957570+0000", 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5520000000000005e-06 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965685+0000", 2024-09-16T02:15:48.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0081150130000000008 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.965988+0000", 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00030228299999999998 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.968839+0000", 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028510089999999998 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977674+0000", 2024-09-16T02:15:48.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088351860000000001 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980727+0000", 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030533589999999998 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.980920+0000", 2024-09-16T02:15:48.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019316299999999999 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:656 3.8s0 3:177f2fba:::smithi19530025-45:head [append 0~0] snapc 14=[12,d] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 45.733441493999997, 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034162017000000003, 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.742 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 656 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957582+0000", 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957583+0000", 2024-09-16T02:15:48.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2600000000000002e-07 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957586+0000", 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1870000000000001e-06 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957595+0000", 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0189999999999995e-06 2024-09-16T02:15:48.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.957602+0000", 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1629999999999999e-06 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972054+0000", 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014452242000000001 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.972340+0000", 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028568300000000001 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.974084+0000", 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001744034 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.977813+0000", 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037288239999999999 2024-09-16T02:15:48.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991646+0000", 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013832947 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:02.991744+0000", 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8292e-05 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.749 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:792 3.8s0 3:1cf09812:::smithi19530025-15:head [append 0~540672] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.686450258000001, 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022538648000000001, 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.750 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 792 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:48.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004573+0000", 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.004576+0000", 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.486e-06 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005623+0000", 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001047302 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005653+0000", 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9495000000000001e-05 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005668+0000", 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4806e-05 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.005825+0000", 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015700299999999999 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.006087+0000", 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026202199999999999 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.755 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.008466+0000", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002379777 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.012994+0000", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045274139999999996 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.026934+0000", 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013940697 2024-09-16T02:15:48.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.027112+0000", 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000177646 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:796 3.8s0 3:1cf09812:::smithi19530025-15:head [setxattr _header (57) in=7b] snapc 1d=[1d,1c,1b,18,d] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.668768499999999, 2024-09-16T02:15:48.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.44076893700000003, 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 796 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:48.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022255+0000", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022257+0000", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.64e-06 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022265+0000", 2024-09-16T02:15:48.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1659999999999994e-06 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022285+0000", 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0225999999999999e-05 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022301+0000", 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5458999999999999e-05 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.029702+0000", 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074008829999999996 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.029842+0000", 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140848 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.030741+0000", 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089882099999999995 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.033161+0000", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024193819999999999 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.462883+0000", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.42972244900000001 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.463024+0000", 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000141063 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.762 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:797 3.8s0 3:1cf09812:::smithi19530025-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.668701703, 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.444497743, 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 797 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022322+0000", 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022323+0000", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4330000000000001e-06 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022330+0000", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4169999999999997e-06 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022350+0000", 2024-09-16T02:15:48.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9729e-05 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.022364+0000", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4735999999999999e-05 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.033962+0000", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011597481999999999 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.034096+0000", 2024-09-16T02:15:48.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013378399999999999 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.463960+0000", 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.42986474299999999 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.464088+0000", 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000127537 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.466820+0000", 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027318820000000001 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s3 91/69 ECSubWrite(tid=183, reqid=osd.11.0:183, at_version=91'176, trim_to=0'0, roll_forward_to=91'170))", 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.713927480999999, 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85248054399999995, 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.768 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977096+0000", 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977098+0000", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7859999999999999e-06 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.977135+0000", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6545999999999999e-05 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827628+0000", 2024-09-16T02:15:48.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.850493583 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827644+0000", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5855999999999999e-05 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827758+0000", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114369 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.827782+0000", 2024-09-16T02:15:48.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3289e-05 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.829489+0000", 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017076529999999999 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.829577+0000", 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7461999999999997e-05 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDScrubReserve(3.8s0,GRANT,e:101,from:0(2),reservation_nonce:0)", 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:30.745208+0000", 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.945815899999999, 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.40676910599999999, 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.745208+0000", 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.745208+0000", 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.745210+0000", 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.917e-06 2024-09-16T02:15:48.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.745219+0000", 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5429999999999999e-06 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.151551+0000", 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.406331363 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.151575+0000", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4779000000000001e-05 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.151864+0000", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028857500000000001 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.151903+0000", 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9045000000000003e-05 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.151977+0000", 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3883999999999999e-05 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.8s0 106/32 ECSubWriteReply(tid=154, last_complete=106'121, committed=1, applied=1))", 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.002791+0000", 2024-09-16T02:15:48.777 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.688232773999999, 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.494499943, 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.002791+0000", 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.002791+0000", 2024-09-16T02:15:48.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.002793+0000", 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.84e-06 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.002804+0000", 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0886e-05 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.496030+0000", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.49322629600000001 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.496043+0000", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3489000000000001e-05 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.496165+0000", 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012112699999999999 2024-09-16T02:15:48.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.497291+0000", 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001126305 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:954 3.8s0 3:1cf09812:::smithi19530025-15:head [read 1709151~633761,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.923423+0000", 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.767600283, 2024-09-16T02:15:48.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.030148145000000001, 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 954 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.782 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923423+0000", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923423+0000", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923428+0000", 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3229999999999999e-06 2024-09-16T02:15:48.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923445+0000", 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7453e-05 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923488+0000", 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2651999999999998e-05 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923505+0000", 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.698e-05 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923665+0000", 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000160032 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923688+0000", 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3413000000000001e-05 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929174+0000", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054859380000000001 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.929464+0000", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029001199999999997 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.953572+0000", 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024107342 2024-09-16T02:15:48.787 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:955 3.8s0 3:1cf09812:::smithi19530025-15:head [read 1855964~486948,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.923550+0000", 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.767474173, 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031911485000000003, 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.788 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 955 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923550+0000", 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923550+0000", 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923551+0000", 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4169999999999999e-06 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923560+0000", 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0890000000000006e-06 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923581+0000", 2024-09-16T02:15:48.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1192999999999999e-05 2024-09-16T02:15:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923596+0000", 2024-09-16T02:15:48.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4769e-05 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923876+0000", 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027968300000000003 2024-09-16T02:15:48.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923897+0000", 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1699e-05 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930965+0000", 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070679840000000002 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.931204+0000", 2024-09-16T02:15:48.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023829999999999999 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.955461+0000", 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024257351 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.797 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:956 3.8s0 3:1cf09812:::smithi19530025-15:head [read 386394~1956518,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.923621+0000", 2024-09-16T02:15:48.798 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.767402785, 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.034729391999999998, 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 956 2024-09-16T02:15:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923621+0000", 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923621+0000", 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923622+0000", 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.33e-06 2024-09-16T02:15:48.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923630+0000", 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.106e-06 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923649+0000", 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8967e-05 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923663+0000", 2024-09-16T02:15:48.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3892e-05 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.923999+0000", 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033562400000000002 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924018+0000", 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9466e-05 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.934455+0000", 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010436956000000001 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.934685+0000", 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023018600000000001 2024-09-16T02:15:48.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.958350+0000", 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.023664865 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.808 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:960 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1261568,checksum,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.924402+0000", 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.766621964, 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036240266, 2024-09-16T02:15:48.809 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 960 2024-09-16T02:15:48.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924402+0000", 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924402+0000", 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924404+0000", 2024-09-16T02:15:48.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0030000000000001e-06 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924415+0000", 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.163e-05 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924439+0000", 2024-09-16T02:15:48.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3952999999999999e-05 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924452+0000", 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2379e-05 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924573+0000", 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012115700000000001 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924591+0000", 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8428999999999999e-05 2024-09-16T02:15:48.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.935810+0000", 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011218934 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.936063+0000", 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025318600000000003 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.960642+0000", 2024-09-16T02:15:48.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024578595000000002 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:961 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1261568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.924481+0000", 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.766542859999999, 2024-09-16T02:15:48.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038140379000000002, 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 961 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924481+0000", 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924481+0000", 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924482+0000", 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4899999999999999e-06 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924491+0000", 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.524e-06 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924507+0000", 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5633000000000001e-05 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924515+0000", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8489999999999995e-06 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924712+0000", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000196808 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924727+0000", 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5206e-05 2024-09-16T02:15:48.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941549+0000", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016821447999999999 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941992+0000", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044276100000000001 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.962621+0000", 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020629660000000001 2024-09-16T02:15:48.823 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:962 3.8s0 3:177f2fba:::smithi19530025-45:head [read 0~1261568,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.924536+0000", 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.766487426999999, 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.041702603999999997, 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:48.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 962 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924536+0000", 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924536+0000", 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924538+0000", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924545+0000", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7360000000000004e-06 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924562+0000", 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6863000000000001e-05 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924572+0000", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0397000000000001e-05 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924871+0000", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029914699999999999 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.924882+0000", 2024-09-16T02:15:48.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0404000000000001e-05 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.950149+0000", 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025267409000000001 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.951455+0000", 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013055129999999999 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:48.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.966239+0000", 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014784392 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:48.829 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:48.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-16T02:15:48.831 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:48.828+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:48.931 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:48.928+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:49.003 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.003 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:49.003 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:49.003 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-16T02:15:49.032 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:49.029+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:49.043 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:49.133 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:49.129+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:49.184 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.184 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:49.184 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:49.184 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:49.184 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-16T02:15:49.234 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:49.229+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:49.323 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 35/32 ECSubWrite(tid=1, reqid=client.4598.0:19, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.041317355999993, 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009183241, 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.324 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279340+0000", 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279341+0000", 2024-09-16T02:15:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2699999999999999e-07 2024-09-16T02:15:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279560+0000", 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021887299999999999 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279644+0000", 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4229999999999998e-05 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279669+0000", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4898000000000001e-05 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279936+0000", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026744200000000001 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.279974+0000", 2024-09-16T02:15:49.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7688000000000002e-05 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288418+0000", 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084440780000000007 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.288523+0000", 2024-09-16T02:15:49.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105405 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=0, reqid=client.4598.0:16, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.037094909999993, 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007088329, 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283563+0000", 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.13e-07 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283754+0000", 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000190578 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283973+0000", 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021948000000000001 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283997+0000", 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3515000000000001e-05 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284288+0000", 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000291174 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.284309+0000", 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0931999999999999e-05 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290581+0000", 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062724 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290651+0000", 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9636999999999998e-05 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.334 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=0, reqid=client.4598.0:31, at_version=35'1, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.024342978000007, 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0097954689999999994, 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296315+0000", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5000000000000002e-07 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296481+0000", 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166086 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296596+0000", 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114555 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296619+0000", 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3218999999999999e-05 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296775+0000", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015605 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296798+0000", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2568000000000001e-05 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306004+0000", 2024-09-16T02:15:49.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0092059859999999993 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.306110+0000", 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010635499999999999 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=1, reqid=client.4598.0:32, at_version=35'2, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.339 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.021678597999994, 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083844969999999994, 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298979+0000", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.298981+0000", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7829999999999999e-06 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299030+0000", 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9014000000000002e-05 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299340+0000", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031047600000000001 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299362+0000", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2118999999999999e-05 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299579+0000", 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000216324 2024-09-16T02:15:49.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299603+0000", 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307264+0000", 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0076608789999999998 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307363+0000", 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9642999999999994e-05 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=2, reqid=client.4598.0:37, at_version=35'3, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.021619377999997, 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085188360000000001, 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.344 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299038+0000", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299039+0000", 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299231+0000", 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192141 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299432+0000", 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020059099999999999 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299455+0000", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3306e-05 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300035+0000", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000580337 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.300057+0000", 2024-09-16T02:15:49.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2099e-05 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307369+0000", 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073119429999999996 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307557+0000", 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018761599999999999 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 35/32 ECSubWrite(tid=2, reqid=client.4598.0:17, at_version=0'0, trim_to=0'0, roll_forward_to=35'2))", 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.021212073000001, 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0079462869999999998, 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:49.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299445+0000", 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299447+0000", 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9630000000000001e-06 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299478+0000", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0175000000000002e-05 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299574+0000", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6416000000000007e-05 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299598+0000", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3578999999999999e-05 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299865+0000", 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026731200000000002 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.299890+0000", 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4892000000000002e-05 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307236+0000", 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073460299999999999 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.307392+0000", 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015592000000000001 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=3, reqid=client.4598.0:38, at_version=35'4, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.019331328000007, 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073537769999999997, 2024-09-16T02:15:49.353 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301326+0000", 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301328+0000", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1900000000000002e-06 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301370+0000", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1734999999999999e-05 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301453+0000", 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2669000000000004e-05 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301474+0000", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1348999999999998e-05 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302441+0000", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096730000000000004 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.302464+0000", 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2921999999999998e-05 2024-09-16T02:15:49.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308614+0000", 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061499950000000001 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308680+0000", 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5617000000000006e-05 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 35/32 ECSubWrite(tid=4, reqid=client.4598.0:40, at_version=35'5, trim_to=0'0, roll_forward_to=0'0))", 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.019055764000001, 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071229939999999997, 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301602+0000", 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301603+0000", 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301769+0000", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166076 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301853+0000", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3932999999999998e-05 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301875+0000", 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1501999999999998e-05 2024-09-16T02:15:49.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303014+0000", 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011396150000000001 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303039+0000", 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4828999999999999e-05 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308686+0000", 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056464779999999999 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.308725+0000", 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9214000000000001e-05 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=45, reqid=client.4598.0:244, at_version=44'36, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:49.362 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.901192534000003, 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013820059000000001, 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419465+0000", 2024-09-16T02:15:49.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419467+0000", 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9400000000000001e-06 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.419693+0000", 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000226312 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431387+0000", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011694154999999999 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431404+0000", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6945000000000001e-05 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431560+0000", 2024-09-16T02:15:49.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015586199999999999 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.431575+0000", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4979e-05 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433182+0000", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016072700000000001 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.433285+0000", 2024-09-16T02:15:49.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010259600000000001 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 44/32 ECSubWrite(tid=48, reqid=client.4598.0:247, at_version=44'40, trim_to=0'0, roll_forward_to=44'35))", 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.885636325999997, 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29780827999999998, 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435021+0000", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435023+0000", 2024-09-16T02:15:49.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.463e-06 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.435063+0000", 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0126000000000002e-05 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731424+0000", 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29636148400000001 2024-09-16T02:15:49.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731439+0000", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.453e-05 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731595+0000", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015660699999999999 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.731614+0000", 2024-09-16T02:15:49.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8243000000000001e-05 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732758+0000", 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011446360000000001 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:33.732829+0000", 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1191000000000003e-05 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 45/32 ECSubWrite(tid=33, reqid=osd.3.0:33, at_version=45'30, trim_to=0'0, roll_forward_to=44'28))", 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 74.963763458000003, 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080490270000000003, 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356894+0000", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356895+0000", 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-09-16T02:15:49.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.356948+0000", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3032000000000003e-05 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357075+0000", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012766899999999999 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357100+0000", 2024-09-16T02:15:49.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5086e-05 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357282+0000", 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018186299999999999 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.357310+0000", 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7348999999999999e-05 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364849+0000", 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075397019999999997 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.364943+0000", 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3709999999999996e-05 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.376 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.1s2 45/32 ECSubWrite(tid=51, reqid=osd.11.0:51, at_version=45'42, trim_to=0'0, roll_forward_to=44'40))", 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 74.959012064999996, 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070518059999999999, 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361645+0000", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361646+0000", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361693+0000", 2024-09-16T02:15:49.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6882000000000003e-05 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361805+0000", 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112046 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.361834+0000", 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9215999999999999e-05 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362268+0000", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043351899999999999 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.362294+0000", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6142000000000001e-05 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368607+0000", 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063134869999999996 2024-09-16T02:15:49.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.368697+0000", 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9844000000000002e-05 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 91/32 ECSubWrite(tid=46, reqid=client.4598.0:825, at_version=91'27, trim_to=0'0, roll_forward_to=84'25))", 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:49.381 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.322654678999999, 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063829200000000003, 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998003+0000", 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998004+0000", 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5600000000000004e-07 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:19.998976+0000", 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00097227800000000003 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001818+0000", 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028420189999999999 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.001830+0000", 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.241e-05 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.002052+0000", 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221108 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.002067+0000", 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5033e-05 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004336+0000", 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022693420000000001 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.004386+0000", 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9774000000000002e-05 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.6s6 91/32 ECSubWrite(tid=51, reqid=client.4598.0:831, at_version=91'33, trim_to=0'0, roll_forward_to=91'28))", 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 29.311274235999999, 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038355847999999998, 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.386 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009383+0000", 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.387 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009385+0000", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3230000000000001e-06 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.009414+0000", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9739000000000001e-05 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.045925+0000", 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036510501000000001 2024-09-16T02:15:49.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.045940+0000", 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5366000000000002e-05 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.046104+0000", 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016406299999999999 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.046141+0000", 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6924000000000001e-05 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.047694+0000", 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015532969999999999 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:20.047739+0000", 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4635000000000002e-05 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.390 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 102/83 ECSubWrite(tid=220, reqid=client.4598.0:914, at_version=102'60, trim_to=0'0, roll_forward_to=99'58))", 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:30.949351+0000", 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 18.371306671999999, 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069879959999999998, 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949351+0000", 2024-09-16T02:15:49.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949351+0000", 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949352+0000", 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3e-07 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949707+0000", 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00035533800000000002 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949822+0000", 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011540900000000001 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.949848+0000", 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5846e-05 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.950021+0000", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017251199999999999 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.950046+0000", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5122000000000001e-05 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.956285+0000", 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0062392919999999996 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:30.956339+0000", 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3746999999999999e-05 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.bs3 103/83 ECSubWrite(tid=228, reqid=osd.7.0:228, at_version=103'67, trim_to=0'0, roll_forward_to=102'65))", 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:31.937989+0000", 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 17.382668953, 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075284050000000002, 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.937989+0000", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.937989+0000", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.937989+0000", 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9400000000000005e-07 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.938050+0000", 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1221000000000002e-05 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.938177+0000", 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012658 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.938202+0000", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4638999999999999e-05 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.938350+0000", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014862600000000001 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.938362+0000", 2024-09-16T02:15:49.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1612e-05 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.945432+0000", 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0070705589999999997 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:31.945517+0000", 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4573999999999994e-05 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 107/32 ECSubWrite(tid=139, reqid=client.4598.0:951, at_version=107'136, trim_to=0'0, roll_forward_to=106'132))", 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.930683+0000", 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.389974755000001, 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014966847, 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930683+0000", 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930683+0000", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930683+0000", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7999999999999996e-07 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.930735+0000", 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1576000000000002e-05 2024-09-16T02:15:49.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.937600+0000", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068656150000000003 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.937629+0000", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8093e-05 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938053+0000", 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042421800000000002 2024-09-16T02:15:49.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938078+0000", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5358999999999999e-05 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.945572+0000", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074934249999999997 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.945650+0000", 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8081e-05 2024-09-16T02:15:49.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 107/32 ECSubWrite(tid=140, reqid=client.4598.0:952, at_version=107'137, trim_to=0'0, roll_forward_to=106'132))", 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.938628+0000", 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.382029247, 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085130999999999991, 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938628+0000", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938628+0000", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938630+0000", 2024-09-16T02:15:49.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2139999999999999e-06 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938678+0000", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7531999999999998e-05 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938948+0000", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026978199999999999 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.939848+0000", 2024-09-16T02:15:49.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090053500000000005 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.940432+0000", 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00058414600000000001 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.940457+0000", 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4199999999999999e-05 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.947047+0000", 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065898809999999997 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.947141+0000", 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4809999999999995e-05 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.408 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 107/32 ECSubWrite(tid=141, reqid=client.4598.0:953, at_version=107'138, trim_to=0'0, roll_forward_to=106'132))", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.938687+0000", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.381970301000001, 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085184590000000008, 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938687+0000", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938687+0000", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938688+0000", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4600000000000003e-07 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938723+0000", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5049000000000003e-05 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941842+0000", 2024-09-16T02:15:49.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031192519999999999 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.941876+0000", 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3998999999999998e-05 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.942276+0000", 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000399782 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.942300+0000", 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3499999999999999e-05 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.947151+0000", 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048513760000000001 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.947206+0000", 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4555000000000002e-05 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.412 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.7s2 107/32 ECSubWrite(tid=142, reqid=osd.3.0:142, at_version=107'140, trim_to=0'0, roll_forward_to=106'132))", 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:35.938728+0000", 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.381929216, 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010075396, 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938728+0000", 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938728+0000", 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938729+0000", 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-07 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.938766+0000", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7431999999999997e-05 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.943223+0000", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044563110000000001 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.943244+0000", 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1613000000000001e-05 2024-09-16T02:15:49.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.943842+0000", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059778900000000005 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.943868+0000", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6313e-05 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.948716+0000", 2024-09-16T02:15:49.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048480099999999998 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:35.948804+0000", 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7330999999999994e-05 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.417 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.418 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 13, 14, 15, 11] out_osds: [10] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 14, 3, 7, 11, 15, 10] 2024-09-16T02:15:49.418 INFO:tasks.thrashosds.thrasher:choose_action: min_in 8 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-16T02:15:49.418 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-16T02:15:49.418 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-16T02:15:49.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T02:15:49.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-16T02:15:49.420 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:49.411+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:49.520 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:49.518+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:49.588 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.588 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:49.588 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-09-16T02:15:49.589 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-16T02:15:49.621 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:49.621+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:49.722 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:49.722+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:49.726 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.727 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-09-16T02:15:49.727 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-09-16T02:15:49.727 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-09-16T02:15:49.727 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.740 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-16T02:15:49.823 INFO:tasks.ceph.osd.6.smithi177.stderr:2024-09-16T02:15:49.821+0000 7f3cf3bc4640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26345) UID: 0 2024-09-16T02:15:49.861 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-09-16T02:15:49.861 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 103.78659587200001, 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.29085150999999998, 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.862 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 1 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071262+0000", 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071265+0000", 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4370000000000001e-06 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071279+0000", 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4229e-05 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071326+0000", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7052999999999998e-05 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071344+0000", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7657999999999999e-05 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071505+0000", 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016140499999999999 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.071522+0000", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6816e-05 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358552+0000", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.28703025900000001 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.358835+0000", 2024-09-16T02:15:49.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028281600000000002 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.359198+0000", 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00036269299999999998 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361174+0000", 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019759809999999999 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:49.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.361926+0000", 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075263000000000001 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362012+0000", 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6162999999999995e-05 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.362114+0000", 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010137 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.868 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4215.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 102.94861567300001, 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.35493744599999999, 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4215", 2024-09-16T02:15:49.869 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/2261801196", 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 124 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909242+0000", 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909243+0000", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3699999999999999e-07 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909249+0000", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.699e-06 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909262+0000", 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2899e-05 2024-09-16T02:15:49.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.909270+0000", 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6729999999999992e-06 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925800+0000", 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016530008999999998 2024-09-16T02:15:49.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.925942+0000", 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014213 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.926148+0000", 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206014 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-09-16T02:15:49.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:06.927483+0000", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00133471 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264017+0000", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.33653442700000002 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264104+0000", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7460000000000006e-05 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:07.264180+0000", 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5488000000000006e-05 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:3 3.3s0 3:c481b30a:::smithi19530025-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.580044494999996, 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.031409012, 2024-09-16T02:15:49.875 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277814+0000", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277815+0000", 2024-09-16T02:15:49.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.57e-06 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277823+0000", 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277847+0000", 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3756000000000001e-05 2024-09-16T02:15:49.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.277865+0000", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8386e-05 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286266+0000", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0084010590000000007 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.286488+0000", 2024-09-16T02:15:49.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221437 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301441+0000", 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014952699999999999 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.301671+0000", 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230576 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309223+0000", 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0075514750000000002 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:28 3.2s0 3:45ca573f:::smithi19530025-10:head [append 0~163840,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.576566102000001, 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.028076635999999999, 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.881 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 28 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281292+0000", 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.281313+0000", 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0729000000000001e-05 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282811+0000", 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014978719999999999 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282846+0000", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5151999999999998e-05 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282872+0000", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6205000000000001e-05 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283375+0000", 2024-09-16T02:15:49.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050260099999999998 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283395+0000", 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0369000000000001e-05 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.289408+0000", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0060129379999999998 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.290377+0000", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096915 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.293023+0000", 2024-09-16T02:15:49.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026455269999999999 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303522+0000", 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010499221 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309176+0000", 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056540100000000001 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.309369+0000", 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000192862 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:29 3.2s0 3:45ca573f:::smithi19530025-10:head [setxattr _header (56) in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.574923092999995, 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02767849, 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.888 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 29 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282935+0000", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282938+0000", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5459999999999998e-06 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282947+0000", 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6039999999999999e-06 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.282968+0000", 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1212e-05 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283002+0000", 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3704999999999997e-05 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283548+0000", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054593199999999995 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283562+0000", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4195999999999999e-05 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.294847+0000", 2024-09-16T02:15:49.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011284812 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.295114+0000", 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000267445 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.296347+0000", 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232517 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.303525+0000", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071784459999999998 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310404+0000", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0068782679999999999 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.310614+0000", 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020980700000000001 2024-09-16T02:15:49.894 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:30 3.2s0 3:45ca573f:::smithi19530025-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.574826873999996, 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038286987000000001, 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.895 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283031+0000", 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283033+0000", 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:49.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283041+0000", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0360000000000006e-06 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283066+0000", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5605e-05 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283082+0000", 2024-09-16T02:15:49.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6010000000000001e-05 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283627+0000", 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054411900000000003 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.283639+0000", 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2245e-05 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297000+0000", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013361519000000001 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.297223+0000", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022285299999999999 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313647+0000", 2024-09-16T02:15:49.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016423941000000001 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.313824+0000", 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017716499999999999 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.321318+0000", 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007493904 2024-09-16T02:15:49.901 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:60 3.2s0 3:431ee5ca:::smithi19530025-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e35)", 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 85.541988453000002, 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025649649, 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.902 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 60 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315870+0000", 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315871+0000", 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.533e-06 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315878+0000", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.216e-06 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315896+0000", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8009e-05 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.315923+0000", 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6602e-05 2024-09-16T02:15:49.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328626+0000", 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012703056000000001 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.328749+0000", 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012296900000000001 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339567+0000", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010818168 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.339766+0000", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019845799999999999 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:24.341519+0000", 2024-09-16T02:15:49.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017536380000000001 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.3s0 45/32 ECSubWriteReply(tid=193, last_complete=45'20, committed=1, applied=1))", 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.490252706000007, 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67552626299999996, 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.908 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367605+0000", 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367607+0000", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.59e-06 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.367617+0000", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9559999999999992e-06 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041131+0000", 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67351448000000003 2024-09-16T02:15:49.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041145+0000", 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3473e-05 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.041306+0000", 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016099500000000001 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:35.043132+0000", 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018257690000000001 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 45/32 ECSubWriteReply(tid=194, last_complete=45'30, committed=1, applied=1))", 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 75.485965026000002, 2024-09-16T02:15:49.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.58165347300000003, 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371893+0000", 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371894+0000", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.371902+0000", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8490000000000006e-06 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951507+0000", 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.57960485299999998 2024-09-16T02:15:49.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951522+0000", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4756999999999999e-05 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.951684+0000", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000162354 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:14:34.953547+0000", 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00186236 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.915 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 79/66 ECSubWrite(tid=150, reqid=client.4598.0:719, at_version=0'0, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 41.791410462999998, 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27886435399999998, 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066448+0000", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066449+0000", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.435e-06 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.066471+0000", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1526999999999999e-05 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343774+0000", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.27730348599999999 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343789+0000", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4946000000000001e-05 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.343981+0000", 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019161199999999999 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.344000+0000", 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9545e-05 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345225+0000", 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012244479999999999 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:08.345312+0000", 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7355000000000006e-05 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 84/83 ECSubWriteReply(tid=304, last_complete=84'106, committed=1, applied=1))", 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 36.694301928999998, 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67482556900000001, 2024-09-16T02:15:49.920 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163556+0000", 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163558+0000", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3200000000000001e-06 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.163565+0000", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5129999999999999e-06 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837152+0000", 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.67358686899999998 2024-09-16T02:15:49.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837166+0000", 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3685e-05 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.837280+0000", 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114374 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:13.838382+0000", 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011018079999999999 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWrite(3.4s6 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 35.764426651999997, 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59632199500000005, 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.924 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093431+0000", 2024-09-16T02:15:49.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093434+0000", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.39e-06 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.093480+0000", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5844999999999999e-05 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688435+0000", 2024-09-16T02:15:49.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59495508799999997 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688450+0000", 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5345999999999998e-05 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688617+0000", 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000166608 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.688648+0000", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1282000000000003e-05 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689677+0000", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010285520000000001 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:14.689753+0000", 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6884000000000005e-05 2024-09-16T02:15:49.928 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:868 3.3s0 3:c628ba0c:::smithi19530025-27:head [copy-from ver 49] snapc 23=[23,22,21,20,1f,1e,1d,18,d] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.935108478, 2024-09-16T02:15:49.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042527884000000002, 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 868 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922750+0000", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922753+0000", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.937e-06 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922766+0000", 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3849000000000001e-05 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922800+0000", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3325999999999999e-05 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922818+0000", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7782e-05 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922967+0000", 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000149605 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922980+0000", 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3159999999999999e-05 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931241+0000", 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082608520000000008 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931506+0000", 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026437699999999999 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.957452+0000", 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025946005000000001 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.962422+0000", 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049703719999999998 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.965127+0000", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027051079999999999 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.965278+0000", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015051100000000001 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:869 3.3s0 3:c628ba0c:::smithi19530025-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e98)", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:49.935 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.934993162000001, 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.038142744999999999, 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 869 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922865+0000", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922867+0000", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9039999999999999e-06 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922875+0000", 2024-09-16T02:15:49.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9659999999999996e-06 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922896+0000", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1433e-05 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.922913+0000", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6415000000000001e-05 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.923098+0000", 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000185014 2024-09-16T02:15:49.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting_for_map not empty", 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.923107+0000", 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6560000000000004e-06 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.931923+0000", 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.008816127 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for blocked object", 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.932051+0000", 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012747799999999999 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.960518+0000", 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.02846655 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.960769+0000", 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025148100000000001 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:26.961008+0000", 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023872100000000001 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDScrubReserve(3.2s0,GRANT,e:108,from:14(4),reservation_nonce:0)", 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 12.615266473, 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.698473758, 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242592+0000", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242593+0000", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8169999999999999e-06 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.242604+0000", 2024-09-16T02:15:49.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0285999999999999e-05 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940227+0000", 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.69762373099999997 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940241+0000", 2024-09-16T02:15:49.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3616e-05 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940507+0000", 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026620000000000002 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.940560+0000", 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2621000000000003e-05 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.941065+0000", 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050548699999999999 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:968 3.2s0 3:45ca573f:::smithi19530025-10:head [stat,delete] snapc 27=[26,25,21,20,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.930241682, 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047565187000000002, 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.946 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 968 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927616+0000", 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927619+0000", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8229999999999999e-06 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927630+0000", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1229999999999999e-05 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927663+0000", 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2669000000000002e-05 2024-09-16T02:15:49.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927680+0000", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6796e-05 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927844+0000", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016416600000000001 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.927861+0000", 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6426000000000002e-05 2024-09-16T02:15:49.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.931526+0000", 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003664984 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.931809+0000", 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028384500000000002 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.934082+0000", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022728620000000001 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.938022+0000", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039393320000000003 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.974917+0000", 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036895178000000001 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.975182+0000", 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026487599999999999 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 109/83 ECSubWriteReply(tid=386, last_complete=109'165, committed=1, applied=1))", 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.872593413000001, 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057427276999999999, 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.952 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985265+0000", 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4300000000000002e-07 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:37.985270+0000", 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0570000000000003e-06 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040852+0000", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055582661999999998 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.040867+0000", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4450000000000001e-05 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.042636+0000", 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001769561 2024-09-16T02:15:49.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.042692+0000", 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5804000000000003e-05 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "MOSDECSubOpWriteReply(3.2s0 109/83 ECSubWriteReply(tid=389, last_complete=109'169, committed=1, applied=1))", 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.809621561, 2024-09-16T02:15:49.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.116722672, 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048237+0000", 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048238+0000", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.863e-06 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.048251+0000", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2556e-05 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162830+0000", 2024-09-16T02:15:49.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.114579261 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162849+0000", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8564e-05 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.164639+0000", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017899229999999999 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.164959+0000", 2024-09-16T02:15:49.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032050500000000001 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:984 3.2s0 3:476b4430:::smithi19530025-30:head [setxattr _header (57) in=7b] snapc 27=[26,25,21,20,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.700089584000001, 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032437222000000002, 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.960 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 984 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:49.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157769+0000", 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157771+0000", 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1799999999999999e-06 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157781+0000", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0013000000000001e-05 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157799+0000", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7853e-05 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157809+0000", 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0849000000000001e-05 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162247+0000", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044376770000000001 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.162526+0000", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000278869 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_started", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.163578+0000", 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010525129999999999 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_committed", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165178+0000", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001599105 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.189999+0000", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024821288 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.190206+0000", 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000206875 2024-09-16T02:15:49.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.4598.0:985 3.2s0 3:476b4430:::smithi19530025-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e109)", 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.700020501999999, 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035613218000000002, 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "continuous": false, 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.4598", 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:49.966 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 985 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157838+0000", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157839+0000", 2024-09-16T02:15:49.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.279e-06 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157845+0000", 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0229999999999998e-06 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157864+0000", 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9317000000000002e-05 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-09-16T02:15:49.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.157877+0000", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3149000000000001e-05 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165341+0000", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074638450000000002 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for rw locks", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.165514+0000", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017282800000000001 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-09-16T02:15:49.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.191138+0000", 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.025624257000000001 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.191391+0000", 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025285299999999999 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-09-16T02:15:38.193451+0000", 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020596669999999998 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.970 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-09-16T02:15:49.971 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-09-16T02:15:49.971 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-09-16T02:15:49.972 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-16T02:15:49.973 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:49.967+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:50.062 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T02:15:50.073 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:50.067+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:50.105 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:50.105 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:50.105 INFO:teuthology.orchestra.run.smithi177.stdout: "num_ops": 0 2024-09-16T02:15:50.106 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:50.120 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-16T02:15:50.174 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:50.173+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:50.236 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:50.236 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [], 2024-09-16T02:15:50.236 INFO:teuthology.orchestra.run.smithi177.stdout: "complaint_time": 30, 2024-09-16T02:15:50.236 INFO:teuthology.orchestra.run.smithi177.stdout: "num_blocked_ops": 0 2024-09-16T02:15:50.236 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:50.250 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-16T02:15:50.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-16T02:15:50.275 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:50.267+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:50.376 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:50.373+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:50.415 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-09-16T02:15:50.415 INFO:teuthology.orchestra.run.smithi177.stdout: "size": 20, 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 600, 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "ops": [ 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4293.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 102.221844343, 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66677636500000004, 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4293", 2024-09-16T02:15:50.416 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.3:0/2460438938", 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 2 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190532+0000", 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.417 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190541+0000", 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9490000000000002e-06 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190553+0000", 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2269999999999999e-05 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.418 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190575+0000", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2381000000000001e-05 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190591+0000", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6169000000000001e-05 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190731+0000", 2024-09-16T02:15:50.419 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013952999999999999 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "waiting for peered", 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.190747+0000", 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5673000000000001e-05 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857073+0000", 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.66632612999999996 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.420 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:08.857308+0000", 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023526299999999999 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 44/32 ECSubWrite(tid=30, reqid=osd.5.0:29, at_version=0'0, trim_to=0'0, roll_forward_to=44'20))", 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.069252911000007, 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0095323660000000004, 2024-09-16T02:15:50.421 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.422 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343123+0000", 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343124+0000", 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.06e-06 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343143+0000", 2024-09-16T02:15:50.423 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8842e-05 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343322+0000", 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000179414 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343345+0000", 2024-09-16T02:15:50.424 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2629e-05 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343569+0000", 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00022398900000000001 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343596+0000", 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6489e-05 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.425 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352536+0000", 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0089404250000000001 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352656+0000", 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000119518 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.426 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 44/32 ECSubWrite(tid=19, reqid=osd.6.0:19, at_version=44'14, trim_to=0'0, roll_forward_to=42'12))", 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.069163664000001, 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0083893449999999994, 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.427 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343212+0000", 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343214+0000", 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5430000000000001e-06 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.428 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343264+0000", 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0175e-05 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343378+0000", 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000114159 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343396+0000", 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7926000000000001e-05 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.429 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343513+0000", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000117056 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.343528+0000", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4324999999999999e-05 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351501+0000", 2024-09-16T02:15:50.430 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079735710000000005 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.351602+0000", 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00010059 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 44/32 ECSubWrite(tid=188, reqid=osd.13.0:188, at_version=44'16, trim_to=0'0, roll_forward_to=43'14))", 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:50.431 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 77.067828753000001, 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0082631800000000002, 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.432 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344547+0000", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344549+0000", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7430000000000001e-06 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344608+0000", 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8953999999999998e-05 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.433 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344655+0000", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7268999999999998e-05 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344677+0000", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1382e-05 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344840+0000", 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016295099999999999 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.434 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.344865+0000", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5035e-05 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352714+0000", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0078496989999999999 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:33.352810+0000", 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.6146999999999999e-05 2024-09-16T02:15:50.435 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.8s5 45/32 ECSubWrite(tid=32, reqid=osd.5.0:31, at_version=0'0, trim_to=0'0, roll_forward_to=45'22))", 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 76.065013406999995, 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.009462359, 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.436 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347363+0000", 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347364+0000", 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.623e-06 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.437 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347386+0000", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1667000000000001e-05 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347450+0000", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.4116999999999996e-05 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.347464+0000", 2024-09-16T02:15:50.438 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.4347e-05 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348099+0000", 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00063462999999999996 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.348122+0000", 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2866000000000001e-05 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.439 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356742+0000", 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086204220000000008 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.356825+0000", 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2687000000000003e-05 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.440 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 45/32 ECSubWrite(tid=28, reqid=osd.7.0:28, at_version=45'10, trim_to=0'0, roll_forward_to=36'8))", 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 76.061103528999993, 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.010034855000000001, 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.441 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351273+0000", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.2600000000000001e-07 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351322+0000", 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.8242000000000003e-05 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.442 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351430+0000", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108572 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351458+0000", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8229e-05 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351962+0000", 2024-09-16T02:15:50.443 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00050377899999999999 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.351989+0000", 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6322e-05 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361119+0000", 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091308310000000007 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.444 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:34.361307+0000", 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000188054 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:394 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 69.505055569000007, 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094620629999999997, 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.445 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 394 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.446 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907320+0000", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907323+0000", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.5399999999999998e-06 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907336+0000", 2024-09-16T02:15:50.447 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.2605999999999999e-05 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907364+0000", 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8538999999999999e-05 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907387+0000", 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2979000000000001e-05 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.448 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907499+0000", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011208 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907666+0000", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016685800000000001 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.909169+0000", 2024-09-16T02:15:50.449 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0015026659999999999 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914888+0000", 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0057194580000000002 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916546+0000", 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001657911 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.450 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.916783+0000", 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000236426 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:395 3.fs0 3:f97ec54c:::smithi19530025-26:head [rollback 8] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 69.504951258999995, 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011190179, 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.451 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 395 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.452 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907425+0000", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907426+0000", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.393e-06 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907433+0000", 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8329999999999996e-06 2024-09-16T02:15:50.453 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907453+0000", 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0176000000000001e-05 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907485+0000", 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.2066000000000001e-05 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.454 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910164+0000", 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0026789510000000002 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.910300+0000", 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00013539300000000001 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.911564+0000", 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001264577 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.455 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.917774+0000", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0062099729999999997 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918506+0000", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00073229000000000002 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918615+0000", 2024-09-16T02:15:50.456 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000108527 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:396 3.fs0 3:f97ec54c:::smithi19530025-26:head [append 0~0] snapc b=[b,a,8] ondisk+write+known_if_redirected+supports_pool_eio e52)", 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 69.504865205000002, 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011749021, 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.457 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 396 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.458 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907511+0000", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907512+0000", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.1060000000000001e-06 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907519+0000", 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.917e-06 2024-09-16T02:15:50.459 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907537+0000", 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7915999999999999e-05 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.907552+0000", 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5396e-05 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.460 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912502+0000", 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0049500600000000001 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.912741+0000", 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00023893499999999999 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.914057+0000", 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0013153119999999999 2024-09-16T02:15:50.461 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.918884+0000", 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0048277240000000003 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919157+0000", 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00027306100000000001 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.462 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:14:40.919260+0000", 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000102594 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 79/66 ECSubWrite(tid=139, reqid=osd.7.0:139, at_version=79'54, trim_to=0'0, roll_forward_to=78'52))", 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.420530479999996, 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0091235069999999995, 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.463 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991846+0000", 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.464 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991847+0000", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.0699999999999999e-06 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.991900+0000", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.3204000000000001e-05 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992025+0000", 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012476300000000001 2024-09-16T02:15:50.465 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992053+0000", 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.8085e-05 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992229+0000", 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017621299999999999 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.466 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:07.992255+0000", 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6319e-05 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000887+0000", 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086322410000000006 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.000969+0000", 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.1612000000000004e-05 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.467 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(osd.7.22:3 3.fs0 3:f2f6c7ec:::smithi19530025-24:head [assert-version v21,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 42.37154348, 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0090619859999999993, 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "started", 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.468 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040833+0000", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040836+0000", 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.4659999999999999e-06 2024-09-16T02:15:50.469 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040855+0000", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8689e-05 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040906+0000", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1381000000000002e-05 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.040930+0000", 2024-09-16T02:15:50.470 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3739000000000001e-05 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041122+0000", 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000191669 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.041284+0000", 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000162917 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.471 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:08.049895+0000", 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0086101249999999997 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 85/66 ECSubWrite(tid=176, reqid=client.4598.0:764, at_version=85'61, trim_to=0'0, roll_forward_to=79'55))", 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 36.318885493000003, 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50299762999999997, 2024-09-16T02:15:50.472 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093491+0000", 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.473 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093493+0000", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2400000000000002e-06 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.093538+0000", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.4971e-05 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595224+0000", 2024-09-16T02:15:50.474 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.50168629200000003 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595240+0000", 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.56e-05 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595366+0000", 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00012640000000000001 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.475 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.595382+0000", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5936000000000001e-05 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596420+0000", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0010379499999999999 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:14.596488+0000", 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 6.8240999999999999e-05 2024-09-16T02:15:50.476 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.3s1 103/83 ECSubWrite(tid=356, reqid=osd.13.0:356, at_version=103'25, trim_to=0'0, roll_forward_to=98'23))", 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 18.468134789, 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0079670350000000008, 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.477 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944241+0000", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944242+0000", 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.7599999999999997e-07 2024-09-16T02:15:50.478 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.944284+0000", 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1752000000000003e-05 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.946707+0000", 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0024230660000000002 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.479 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.946731+0000", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4774999999999999e-05 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.947311+0000", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00057995599999999996 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.947338+0000", 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6803000000000001e-05 2024-09-16T02:15:50.480 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.952113+0000", 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0047752300000000001 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:31.952208+0000", 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.4877000000000007e-05 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.481 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "osd_op(client.4598.0:957 3.fs0 3:f560586e:::smithi19530025-12:head [create,setxattr clNOgfqtsg\\aUXcbnbsgicte[dlcgscTctZg\\UYb (1261) in=40b,setxattr ^kePZO^jc[`nRrZQr^g`hWUbUbmVa_[^`ndQ_ebi (207) in=40b,setxattr pd\\tReSgkdQqSk_oQZgRskailqSfXkleakRgOpgs (735) in=40b,setxattr `\\Uhenhpc\\oVlscgQb^Wpj]b[jPkrs]S`tY^nfNX (1506) in=40b,setxattr mYhN^Rig]VasbVqh]mfR[[pqUcOYh]_aWp\\m_tls (1891) in=40b,setxattr hie`]lstsZtOilXYaRecaiXtt]gcmm^a[tWjt[YT (311) in=40b,setxattr hkRZhYOPaigVYZmnZe^p`NTbkSNSiVgfgVijksca (455) in=40b,setxattr `sdWbhfe[ng_eOr]YcXa]ahtbPgochac``jdt`hh (1262) in=40b,setxattr aPlreWi_Xm]iktYWh`g_aSaRPrXe]adtOdO]]bsq (347) in=40b,setxattr p_qb[TbOoSb^Z[jcnVcp[Vb_XQ^gddoWiVa^NPYR (385) in=40b,setxattr VgQd`_b]^jekaYilkQoqdami]]mhVp\\fbWfmkbVU (1295) in=40b,setxattr tWfY]UYjtVla^t`]`Z[s\\sp`]jp^jcoVP[kaYTjb (1893) in=40b,setxattr ZeWXgW`YOnsgbbrr`eRYbpbOQeTm_g^hkeqUdl_X (982) in=40b,setxattr aoh__jnjZmcV_Zd[lTjob_PfRoWV (1822) in=28b,setxattr Og\\PgsWdl_jTZ]kbbRXdO`bhQsdfmNUjgURgh\\`^ (1997) in=40b,setxattr fWcmbPRf_fT[s`o`jq^\\U_m_UlTQhjpPfe]Oq^Zs (1645) in=40b,setxattr jQT^\\Zie\\orcqPOr]sg\\\\PWNdY`OYs^QYktWQsTd (1789) in=40b,setxattr N_TeWQ`\\cU]`mQPYod^P]RXe[YaVQncV^sk`_bVn (1785) in=40b,setxattr jQhOSafV (1822) in=8b,setxattr ]daaodicjtV`q_`__etNiom_n`abiaa`sbrXmcaT (947) in=40b] snapc 27=[27,26,25,21,20,1f,1e,18] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.487740691999999, 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.026264025999999999, 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "client_info": { 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "client": "client.4598", 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "client_addr": "172.21.15.195:0/206821130", 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: "tid": 957 2024-09-16T02:15:50.482 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924635+0000", 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.483 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924639+0000", 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.32e-06 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924738+0000", 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.8863999999999994e-05 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924768+0000", 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 3.0669000000000001e-05 2024-09-16T02:15:50.484 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924786+0000", 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7977000000000001e-05 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924949+0000", 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000162325 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "op must wait for map", 2024-09-16T02:15:50.485 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.924971+0000", 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2079e-05 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.934469+0000", 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0094982409999999993 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "started", 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.934801+0000", 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00033254299999999998 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.486 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.937819+0000", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.003017568 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949768+0000", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011948780000000001 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "commit_sent", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949987+0000", 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00021956999999999999 2024-09-16T02:15:50.487 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.950899+0000", 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00091208999999999995 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=128, reqid=osd.6.0:128, at_version=107'114, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.485405215, 2024-09-16T02:15:50.488 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.017360448000000001, 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926971+0000", 2024-09-16T02:15:50.489 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.926973+0000", 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.72e-06 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927024+0000", 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.1595000000000003e-05 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.490 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927136+0000", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00011198 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927160+0000", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.3462999999999999e-05 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927318+0000", 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00015849200000000001 2024-09-16T02:15:50.491 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.927336+0000", 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.7538999999999999e-05 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944255+0000", 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.016919328000000001 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.492 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944331+0000", 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.6330999999999998e-05 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.4s4 107/66 ECSubWrite(tid=247, reqid=osd.7.0:247, at_version=107'100, trim_to=0'0, roll_forward_to=106'98))", 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.482719965999999, 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014333392, 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.493 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929656+0000", 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.494 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929657+0000", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.7e-07 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.929703+0000", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6354999999999999e-05 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930028+0000", 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000325528 2024-09-16T02:15:50.495 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930050+0000", 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.1165000000000001e-05 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930240+0000", 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000190729 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.496 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.930267+0000", 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6435e-05 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943883+0000", 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.013616453000000001 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.943989+0000", 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000106257 2024-09-16T02:15:50.497 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=129, reqid=client.4598.0:948, at_version=107'116, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.480626253000001, 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.015009710000000001, 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.498 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931750+0000", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931752+0000", 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.0470000000000001e-06 2024-09-16T02:15:50.499 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931799+0000", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.6705000000000001e-05 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931979+0000", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018057500000000001 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932002+0000", 2024-09-16T02:15:50.500 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.2636000000000001e-05 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932166+0000", 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00016461500000000001 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932182+0000", 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.5536000000000002e-05 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.501 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946687+0000", 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014505120999999999 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946760+0000", 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.2475e-05 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.bs5 107/83 ECSubWrite(tid=248, reqid=osd.7.0:248, at_version=107'73, trim_to=0'0, roll_forward_to=106'71))", 2024-09-16T02:15:50.502 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.480491913, 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.014855493000000001, 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.503 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931884+0000", 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931892+0000", 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 7.9189999999999999e-06 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.931934+0000", 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 4.1628999999999997e-05 2024-09-16T02:15:50.504 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932094+0000", 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.000159963 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.932112+0000", 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.8168999999999998e-05 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.505 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.933629+0000", 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.001517357 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.933654+0000", 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4644999999999999e-05 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946650+0000", 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.012995860999999999 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.506 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.946740+0000", 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.9950000000000004e-05 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=130, reqid=client.4598.0:949, at_version=107'117, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.474195784000001, 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011725154, 2024-09-16T02:15:50.507 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938180+0000", 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.508 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938182+0000", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.657e-06 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938232+0000", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.0345e-05 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938407+0000", 2024-09-16T02:15:50.509 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00017511499999999999 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938432+0000", 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.4606000000000001e-05 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944442+0000", 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00600995 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.510 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.944456+0000", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.3814e-05 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949814+0000", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0053581490000000004 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949905+0000", 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.1518e-05 2024-09-16T02:15:50.511 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "description": "MOSDECSubOpWrite(3.ds1 107/32 ECSubWrite(tid=131, reqid=client.4598.0:950, at_version=107'118, trim_to=0'0, roll_forward_to=106'110))", 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "initiated_at": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "age": 14.474134789000001, 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.011753016, 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "continuous": false, 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "type_data": { 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "flag_point": "reached pg", 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: "events": [ 2024-09-16T02:15:50.512 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "initiated", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "header_read", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938241+0000", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "throttled", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938242+0000", 2024-09-16T02:15:50.513 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 9.64e-07 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "all_read", 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938301+0000", 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 5.8535000000000001e-05 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "dispatched", 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938489+0000", 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.00018785399999999999 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.514 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "queued_for_pg", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.938505+0000", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 1.6626e-05 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "reached_pg", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.945894+0000", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0073882510000000002 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_started", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.945920+0000", 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 2.6282999999999999e-05 2024-09-16T02:15:50.515 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "sub_op_committed", 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949914+0000", 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 0.0039943690000000002 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: }, 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: { 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "event": "done", 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "time": "2024-09-16T02:15:35.949994+0000", 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: "duration": 8.0134000000000002e-05 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.516 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.517 INFO:teuthology.orchestra.run.smithi177.stdout: } 2024-09-16T02:15:50.517 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-09-16T02:15:50.517 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-09-16T02:15:50.518 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-09-16T02:15:50.518 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-16T02:15:50.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-09-16T02:15:50.519 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-16T02:15:50.520 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-16T02:15:50.520 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:50.516+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:50.520 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-09-16T02:15:50.520 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-16T02:15:50.521 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-09-16T02:15:50.521 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-16T02:15:50.528 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-09-16T02:15:50.528 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-16T02:15:50.555 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-09-16T02:15:50.555 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-16T02:15:50.583 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-09-16T02:15:50.583 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-16T02:15:50.613 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-09-16T02:15:50.613 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-16T02:15:50.618 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:50.611+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:50.644 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8: {} 2024-09-16T02:15:50.645 INFO:teuthology.orchestra.run.smithi003.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-16T02:15:50.678 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9: {} 2024-09-16T02:15:50.678 INFO:teuthology.orchestra.run.smithi003.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-16T02:15:50.690 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:50.690 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:50.705 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-16T02:15:50.711 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10: {} 2024-09-16T02:15:50.712 INFO:teuthology.orchestra.run.smithi003.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-16T02:15:50.719 INFO:tasks.ceph.osd.7.smithi195.stderr:2024-09-16T02:15:50.713+0000 7f684cb17640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26581) UID: 0 2024-09-16T02:15:50.747 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11: {} 2024-09-16T02:15:50.747 INFO:teuthology.orchestra.run.smithi003.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-16T02:15:50.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.12: {} 2024-09-16T02:15:50.782 INFO:teuthology.orchestra.run.smithi003.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-16T02:15:50.819 INFO:teuthology.orchestra.run.smithi003.stdout:osd.13: {} 2024-09-16T02:15:50.819 INFO:teuthology.orchestra.run.smithi003.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-16T02:15:50.819 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-09-16T02:15:50.818+0000 7f73ce62c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26325) UID: 0 2024-09-16T02:15:50.828 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:50.828 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:50.842 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-16T02:15:50.855 INFO:teuthology.orchestra.run.smithi003.stdout:osd.14: {} 2024-09-16T02:15:50.855 INFO:teuthology.orchestra.run.smithi003.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-16T02:15:50.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-09-16T02:15:50.854+0000 7f5893fff640 0 -- 172.21.15.3:0/4125431882 >> [v2:172.21.15.195:6802/3564981919,v1:172.21.15.195:6803/3564981919] conn(0x7f5894224a10 msgr2=0x7f5894226e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-16T02:15:50.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:15:50.920 INFO:tasks.ceph.osd.14.smithi177.stderr:2024-09-16T02:15:50.916+0000 7f4a62fed640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26349) UID: 0 2024-09-16T02:15:50.962 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:50.962 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:50.977 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-16T02:15:51.021 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:51.018+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:51.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:51.099 INFO:teuthology.orchestra.run.smithi003.stdout:osd.15: {} 2024-09-16T02:15:51.099 INFO:teuthology.orchestra.run.smithi003.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-16T02:15:51.122 INFO:tasks.ceph.osd.2.smithi177.stderr:2024-09-16T02:15:51.120+0000 7f0d56b74640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26343) UID: 0 2024-09-16T02:15:51.148 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.149 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.163 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-16T02:15:51.224 INFO:tasks.ceph.osd.12.smithi003.stderr:2024-09-16T02:15:51.222+0000 7f24102b2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26331) UID: 0 2024-09-16T02:15:51.234 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:15:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:15:51.253 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-16T02:15:50.066241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":114229811,"num_objects":99,"num_object_clones":48,"num_object_copies":673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":6,"num_read":538,"num_read_kb":159792,"num_write":1080,"num_write_kb":156178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1499,"ondisk_log_size":1499,"up":130,"acting":130,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":130,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":15,"kb":1415577600,"kb_used":682540,"kb_used_data":210560,"kb_used_omap":23,"kb_used_meta":471656,"kb_avail":1414895060,"statfs":{"total":1449551462400,"available":1448852541440,"internally_reserved":0,"allocated":215613440,"data_stored":288164573,"data_compressed":4136437,"data_compressed_allocated":5677056,"data_compressed_original":12214272,"omap_allocated":23922,"internal_metadata":482976398},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.043487"},"pg_stats":[{"pgid":"3.f","version":"110'55","reported_seq":439,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.181658+0000","last_change":"2024-09-16T02:15:39.026154+0000","last_active":"2024-09-16T02:15:43.181658+0000","last_peered":"2024-09-16T02:15:43.181658+0000","last_clean":"2024-09-16T02:15:43.181658+0000","last_became_active":"2024-09-16T02:14:22.485412+0000","last_became_peered":"2024-09-16T02:14:22.485412+0000","last_unstale":"2024-09-16T02:15:43.181658+0000","last_undegraded":"2024-09-16T02:15:43.181658+0000","last_fullsized":"2024-09-16T02:15:43.181658+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:15:47.420465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.020148758999999999,"stat_sum":{"num_bytes":4751360,"num_objects":5,"num_object_clones":2,"num_object_copies":35,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":2907,"num_write":38,"num_write_kb":5056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12,4,13,7,6,3],"acting":[2,12,4,13,7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.e","version":"106'79","reported_seq":555,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.631933+0000","last_change":"2024-09-16T02:15:38.989889+0000","last_active":"2024-09-16T02:15:46.631933+0000","last_peered":"2024-09-16T02:15:46.631933+0000","last_clean":"2024-09-16T02:15:46.631933+0000","last_became_active":"2024-09-16T02:15:13.070271+0000","last_became_peered":"2024-09-16T02:15:13.070271+0000","last_unstale":"2024-09-16T02:15:46.631933+0000","last_undegraded":"2024-09-16T02:15:46.631933+0000","last_fullsized":"2024-09-16T02:15:46.631933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'66","last_scrub_stamp":"2024-09-16T02:15:27.680026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.680026+0000","objects_scrubbed":8,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.051994+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00059237999999999999,"stat_sum":{"num_bytes":11616256,"num_objects":6,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":32,"num_read_kb":18552,"num_write":73,"num_write_kb":11344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":19972096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,1,14,11,0,15],"acting":[8,9,1,14,11,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.d","version":"110'138","reported_seq":517,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.182882+0000","last_change":"2024-09-16T02:15:42.761762+0000","last_active":"2024-09-16T02:15:43.182882+0000","last_peered":"2024-09-16T02:15:43.182882+0000","last_clean":"2024-09-16T02:15:43.182882+0000","last_became_active":"2024-09-16T02:14:22.476355+0000","last_became_peered":"2024-09-16T02:14:22.476355+0000","last_unstale":"2024-09-16T02:15:43.182882+0000","last_undegraded":"2024-09-16T02:15:43.182882+0000","last_fullsized":"2024-09-16T02:15:43.182882+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'138","last_scrub_stamp":"2024-09-16T02:15:42.761648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:42.761648+0000","objects_scrubbed":7,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:00.373993+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.027258724000000002,"stat_sum":{"num_bytes":6602752,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":15024,"num_write":62,"num_write_kb":12320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0,12,15,7,3],"acting":[6,2,0,12,15,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.c","version":"99'31","reported_seq":417,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.489032+0000","last_change":"2024-09-16T02:15:38.986870+0000","last_active":"2024-09-16T02:15:46.489032+0000","last_peered":"2024-09-16T02:15:46.489032+0000","last_clean":"2024-09-16T02:15:46.489032+0000","last_became_active":"2024-09-16T02:14:22.483959+0000","last_became_peered":"2024-09-16T02:14:22.483959+0000","last_unstale":"2024-09-16T02:15:46.489032+0000","last_undegraded":"2024-09-16T02:15:46.489032+0000","last_fullsized":"2024-09-16T02:15:46.489032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'29","last_scrub_stamp":"2024-09-16T02:15:27.850068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:27.850068+0000","objects_scrubbed":2,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:50.020929+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0014552569999999999,"stat_sum":{"num_bytes":3162112,"num_objects":1,"num_object_clones":0,"num_object_copies":7,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":7055,"num_write":13,"num_write_kb":3088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[14,7,6,12,4,3,15],"acting":[14,7,6,12,4,3,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":14,"acting_primary":14,"purged_snaps":[]},{"pgid":"3.b","version":"110'82","reported_seq":506,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.442507+0000","last_change":"2024-09-16T02:15:39.014614+0000","last_active":"2024-09-16T02:15:49.442507+0000","last_peered":"2024-09-16T02:15:49.442507+0000","last_clean":"2024-09-16T02:15:49.442507+0000","last_became_active":"2024-09-16T02:15:13.074136+0000","last_became_peered":"2024-09-16T02:15:13.074136+0000","last_unstale":"2024-09-16T02:15:49.442507+0000","last_undegraded":"2024-09-16T02:15:49.442507+0000","last_fullsized":"2024-09-16T02:15:49.442507+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"107'73","last_scrub_stamp":"2024-09-16T02:15:37.080155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.080155+0000","objects_scrubbed":6,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:37.299907+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0.02296316,"stat_sum":{"num_bytes":10289152,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":38,"num_read_kb":12154,"num_write":55,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":8077312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,15,1,9,8,2,13],"acting":[7,15,1,9,8,2,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":390,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.183076+0000","last_change":"2024-09-16T02:15:39.001525+0000","last_active":"2024-09-16T02:15:43.183076+0000","last_peered":"2024-09-16T02:15:43.183076+0000","last_clean":"2024-09-16T02:15:43.183076+0000","last_became_active":"2024-09-16T02:15:13.070217+0000","last_became_peered":"2024-09-16T02:15:13.070217+0000","last_unstale":"2024-09-16T02:15:43.183076+0000","last_undegraded":"2024-09-16T02:15:43.183076+0000","last_fullsized":"2024-09-16T02:15:43.183076+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:37.746545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.746545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:46.187884+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0015715779999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,14,1,9,8,15,2],"acting":[6,14,1,9,8,15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"3.9","version":"109'30","reported_seq":325,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.442474+0000","last_change":"2024-09-16T02:15:38.993755+0000","last_active":"2024-09-16T02:15:49.442474+0000","last_peered":"2024-09-16T02:15:49.442474+0000","last_clean":"2024-09-16T02:15:49.442474+0000","last_became_active":"2024-09-16T02:14:43.948134+0000","last_became_peered":"2024-09-16T02:14:43.948134+0000","last_unstale":"2024-09-16T02:15:49.442474+0000","last_undegraded":"2024-09-16T02:15:49.442474+0000","last_fullsized":"2024-09-16T02:15:49.442474+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'26","last_scrub_stamp":"2024-09-16T02:15:30.066659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:30.066659+0000","objects_scrubbed":2,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:35.439319+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.001923983,"stat_sum":{"num_bytes":3768320,"num_objects":2,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,13,1,0,8,6,12],"acting":[7,13,1,0,8,6,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"109'123","reported_seq":527,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.045404+0000","last_change":"2024-09-16T02:15:38.992081+0000","last_active":"2024-09-16T02:15:49.045404+0000","last_peered":"2024-09-16T02:15:49.045404+0000","last_clean":"2024-09-16T02:15:49.045404+0000","last_became_active":"2024-09-16T02:14:22.484724+0000","last_became_peered":"2024-09-16T02:14:22.484724+0000","last_unstale":"2024-09-16T02:15:49.045404+0000","last_undegraded":"2024-09-16T02:15:49.045404+0000","last_fullsized":"2024-09-16T02:15:49.045404+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:43.704550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001455411,"stat_sum":{"num_bytes":9732096,"num_objects":6,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":24888,"num_write":67,"num_write_kb":17776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0,4,15,2,1],"acting":[5,7,0,4,15,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"26'112","reported_seq":321,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.006182+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:15:46.006182+0000","last_peered":"2024-09-16T02:15:46.006182+0000","last_clean":"2024-09-16T02:15:46.006182+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:15:46.006182+0000","last_undegraded":"2024-09-16T02:15:46.006182+0000","last_fullsized":"2024-09-16T02:15:46.006182+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":193,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.045519+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:15:49.045519+0000","last_peered":"2024-09-16T02:15:49.045519+0000","last_clean":"2024-09-16T02:15:49.045519+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:15:49.045519+0000","last_undegraded":"2024-09-16T02:15:49.045519+0000","last_fullsized":"2024-09-16T02:15:49.045519+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"110'178","reported_seq":625,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.006678+0000","last_change":"2024-09-16T02:15:39.021388+0000","last_active":"2024-09-16T02:15:46.006678+0000","last_peered":"2024-09-16T02:15:46.006678+0000","last_clean":"2024-09-16T02:15:46.006678+0000","last_became_active":"2024-09-16T02:15:13.072281+0000","last_became_peered":"2024-09-16T02:15:13.072281+0000","last_unstale":"2024-09-16T02:15:46.006678+0000","last_undegraded":"2024-09-16T02:15:46.006678+0000","last_fullsized":"2024-09-16T02:15:46.006678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'161","last_scrub_stamp":"2024-09-16T02:15:37.976949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:37.976949+0000","objects_scrubbed":14,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:01.555650+0000","scrub_duration":863,"objects_trimmed":0,"snaptrim_duration":0.026017814,"stat_sum":{"num_bytes":18137088,"num_objects":16,"num_object_clones":9,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":72,"num_read_kb":17166,"num_write":134,"num_write_kb":21728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":8847360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,12,6,4,14,1,0],"acting":[13,12,6,4,14,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":186,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.181461+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:15:43.181461+0000","last_peered":"2024-09-16T02:15:43.181461+0000","last_clean":"2024-09-16T02:15:43.181461+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:15:43.181461+0000","last_undegraded":"2024-09-16T02:15:43.181461+0000","last_fullsized":"2024-09-16T02:15:43.181461+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:25.996964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"110'241","reported_seq":668,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.182328+0000","last_change":"2024-09-16T02:15:39.022506+0000","last_active":"2024-09-16T02:15:43.182328+0000","last_peered":"2024-09-16T02:15:43.182328+0000","last_clean":"2024-09-16T02:15:43.182328+0000","last_became_active":"2024-09-16T02:14:58.940506+0000","last_became_peered":"2024-09-16T02:14:58.940506+0000","last_unstale":"2024-09-16T02:15:43.182328+0000","last_undegraded":"2024-09-16T02:15:43.182328+0000","last_fullsized":"2024-09-16T02:15:43.182328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"99'209","last_scrub_stamp":"2024-09-16T02:15:29.441137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:29.441137+0000","objects_scrubbed":13,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:39.286585+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.022827719,"stat_sum":{"num_bytes":18284544,"num_objects":13,"num_object_clones":7,"num_object_copies":91,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":55,"num_read_kb":17943,"num_write":190,"num_write_kb":25232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":15138816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,14,9,5,6,0,3],"acting":[11,14,9,5,6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"1d","length":"1"},{"start":"1f","length":"1"},{"start":"22","length":"3"},{"start":"26","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.006221+0000","last_change":"2024-09-16T02:14:10.239803+0000","last_active":"2024-09-16T02:15:46.006221+0000","last_peered":"2024-09-16T02:15:46.006221+0000","last_clean":"2024-09-16T02:15:46.006221+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:15:46.006221+0000","last_undegraded":"2024-09-16T02:15:46.006221+0000","last_fullsized":"2024-09-16T02:15:46.006221+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:20.178879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.0","version":"48'8","reported_seq":387,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.338214+0000","last_change":"2024-09-16T02:15:39.006681+0000","last_active":"2024-09-16T02:15:49.338214+0000","last_peered":"2024-09-16T02:15:49.338214+0000","last_clean":"2024-09-16T02:15:49.338214+0000","last_became_active":"2024-09-16T02:14:22.474075+0000","last_became_peered":"2024-09-16T02:14:22.474075+0000","last_unstale":"2024-09-16T02:15:49.338214+0000","last_undegraded":"2024-09-16T02:15:49.338214+0000","last_fullsized":"2024-09-16T02:15:49.338214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:14:21.222693+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:43.918722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082058989999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":7,"num_write_kb":1312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,14,6,1,5,0,15],"acting":[12,14,6,1,5,0,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":196,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.181748+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:15:43.181748+0000","last_peered":"2024-09-16T02:15:43.181748+0000","last_clean":"2024-09-16T02:15:43.181748+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:15:43.181748+0000","last_undegraded":"2024-09-16T02:15:43.181748+0000","last_fullsized":"2024-09-16T02:15:43.181748+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"109'31","reported_seq":482,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.006919+0000","last_change":"2024-09-16T02:15:40.057360+0000","last_active":"2024-09-16T02:15:46.006919+0000","last_peered":"2024-09-16T02:15:46.006919+0000","last_clean":"2024-09-16T02:15:46.006919+0000","last_became_active":"2024-09-16T02:15:13.074572+0000","last_became_peered":"2024-09-16T02:15:13.074572+0000","last_unstale":"2024-09-16T02:15:46.006919+0000","last_undegraded":"2024-09-16T02:15:46.006919+0000","last_fullsized":"2024-09-16T02:15:46.006919+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'31","last_scrub_stamp":"2024-09-16T02:15:40.057220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:40.057220+0000","objects_scrubbed":3,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:54.439810+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0015176969999999999,"stat_sum":{"num_bytes":1687552,"num_objects":3,"num_object_clones":1,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":5328,"num_write":29,"num_write_kb":2080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2949120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,2,11,14,4,5,12],"acting":[13,2,11,14,4,5,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"3.5","version":"110'81","reported_seq":215,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.338421+0000","last_change":"2024-09-16T02:15:42.890394+0000","last_active":"2024-09-16T02:15:49.338421+0000","last_peered":"2024-09-16T02:15:49.338421+0000","last_clean":"2024-09-16T02:15:49.338421+0000","last_became_active":"2024-09-16T02:15:13.073157+0000","last_became_peered":"2024-09-16T02:15:13.073157+0000","last_unstale":"2024-09-16T02:15:49.338421+0000","last_undegraded":"2024-09-16T02:15:49.338421+0000","last_fullsized":"2024-09-16T02:15:49.338421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'81","last_scrub_stamp":"2024-09-16T02:15:42.890094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:42.890094+0000","objects_scrubbed":8,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:00.610183+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.055262523000000001,"stat_sum":{"num_bytes":9093120,"num_objects":8,"num_object_clones":4,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":46,"num_read_kb":33437,"num_write":39,"num_write_kb":13936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5144576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[12,3,15,14,0,7,8],"acting":[12,3,15,14,0,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":12,"acting_primary":12,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.182912+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:15:43.182912+0000","last_peered":"2024-09-16T02:15:43.182912+0000","last_clean":"2024-09-16T02:15:43.182912+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:15:43.182912+0000","last_undegraded":"2024-09-16T02:15:43.182912+0000","last_fullsized":"2024-09-16T02:15:43.182912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"110'106","reported_seq":530,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-16T02:15:49.442576+0000","last_change":"2024-09-16T02:15:39.944949+0000","last_active":"2024-09-16T02:15:49.442576+0000","last_peered":"2024-09-16T02:15:49.442576+0000","last_clean":"2024-09-16T02:15:49.442576+0000","last_became_active":"2024-09-16T02:14:55.933032+0000","last_became_peered":"2024-09-16T02:14:55.933032+0000","last_unstale":"2024-09-16T02:15:49.442576+0000","last_undegraded":"2024-09-16T02:15:49.442576+0000","last_fullsized":"2024-09-16T02:15:49.442576+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'106","last_scrub_stamp":"2024-09-16T02:15:39.944501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:39.944501+0000","objects_scrubbed":7,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:07.530832+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.020947415,"stat_sum":{"num_bytes":4014080,"num_objects":7,"num_object_clones":4,"num_object_copies":49,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":1359,"num_write":93,"num_write_kb":6832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":4112384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,12,11,1,2,15,13],"acting":[7,12,11,1,2,15,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.631706+0000","last_change":"2024-09-16T02:14:10.393398+0000","last_active":"2024-09-16T02:15:46.631706+0000","last_peered":"2024-09-16T02:15:46.631706+0000","last_clean":"2024-09-16T02:15:46.631706+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:15:46.631706+0000","last_undegraded":"2024-09-16T02:15:46.631706+0000","last_fullsized":"2024-09-16T02:15:46.631706+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:21.656463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.7","version":"110'152","reported_seq":526,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:46.544120+0000","last_change":"2024-09-16T02:15:39.398146+0000","last_active":"2024-09-16T02:15:46.544120+0000","last_peered":"2024-09-16T02:15:46.544120+0000","last_clean":"2024-09-16T02:15:46.544120+0000","last_became_active":"2024-09-16T02:14:22.493399+0000","last_became_peered":"2024-09-16T02:14:22.493399+0000","last_unstale":"2024-09-16T02:15:46.544120+0000","last_undegraded":"2024-09-16T02:15:46.544120+0000","last_fullsized":"2024-09-16T02:15:46.544120+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"108'144","last_scrub_stamp":"2024-09-16T02:15:38.304158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:38.304158+0000","objects_scrubbed":10,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:04.664921+0000","scrub_duration":555,"objects_trimmed":0,"snaptrim_duration":0.41430857300000001,"stat_sum":{"num_bytes":9994240,"num_objects":10,"num_object_clones":5,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":26,"num_read_kb":2074,"num_write":84,"num_write_kb":13776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,8,9,7,15,5,14],"acting":[3,8,9,7,15,5,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":197,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:45.924265+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:15:45.924265+0000","last_peered":"2024-09-16T02:15:45.924265+0000","last_clean":"2024-09-16T02:15:45.924265+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:15:45.924265+0000","last_undegraded":"2024-09-16T02:15:45.924265+0000","last_fullsized":"2024-09-16T02:15:45.924265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.6","version":"110'49","reported_seq":440,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-16T02:15:45.924536+0000","last_change":"2024-09-16T02:15:39.006345+0000","last_active":"2024-09-16T02:15:45.924536+0000","last_peered":"2024-09-16T02:15:45.924536+0000","last_clean":"2024-09-16T02:15:45.924536+0000","last_became_active":"2024-09-16T02:14:22.495339+0000","last_became_peered":"2024-09-16T02:14:22.495339+0000","last_unstale":"2024-09-16T02:15:45.924536+0000","last_undegraded":"2024-09-16T02:15:45.924536+0000","last_fullsized":"2024-09-16T02:15:45.924536+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"106'45","last_scrub_stamp":"2024-09-16T02:15:36.983585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:21.222693+0000","last_clean_scrub_stamp":"2024-09-16T02:15:36.983585+0000","objects_scrubbed":3,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:53.480085+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.012849442000000001,"stat_sum":{"num_bytes":2506752,"num_objects":4,"num_object_clones":2,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":1828,"num_write":36,"num_write_kb":2736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,13,1,8,6,2,9],"acting":[15,13,1,8,6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":194,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-09-16T02:15:43.183074+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:15:43.183074+0000","last_peered":"2024-09-16T02:15:43.183074+0000","last_clean":"2024-09-16T02:15:43.183074+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:15:43.183074+0000","last_undegraded":"2024-09-16T02:15:43.183074+0000","last_fullsized":"2024-09-16T02:15:43.183074+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":113639424,"num_objects":95,"num_object_clones":48,"num_object_copies":665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":6,"num_read":452,"num_read_kb":159719,"num_write":933,"num_write_kb":153504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":65617920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":202825728,"data_stored":280113152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},"log_size":1384,"ondisk_log_size":1384,"up":112,"acting":112,"num_store_stats":16},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280536,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47720,"kb_used_data":12372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94324120,"statfs":{"total":96636764160,"available":96587898880,"internally_reserved":0,"allocated":12668928,"data_stored":19342424,"data_compressed":303757,"data_compressed_allocated":413696,"data_compressed_original":1257472,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44552,"kb_used_data":13360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327288,"statfs":{"total":96636764160,"available":96591142912,"internally_reserved":0,"allocated":13680640,"data_stored":19527983,"data_compressed":274262,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280536,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45316,"kb_used_data":9968,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94326524,"statfs":{"total":96636764160,"available":96590360576,"internally_reserved":0,"allocated":10207232,"data_stored":13437609,"data_compressed":268173,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313241,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50048,"kb_used_data":18860,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94321792,"statfs":{"total":96636764160,"available":96585515008,"internally_reserved":0,"allocated":19312640,"data_stored":24339000,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280537,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47140,"kb_used_data":15952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324700,"statfs":{"total":96636764160,"available":96588492800,"internally_reserved":0,"allocated":16334848,"data_stored":22540856,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313239,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48388,"kb_used_data":17204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323452,"statfs":{"total":96636764160,"available":96587214848,"internally_reserved":0,"allocated":17616896,"data_stored":23544376,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41688,"kb_used_data":10500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94330152,"statfs":{"total":96636764160,"available":96594075648,"internally_reserved":0,"allocated":10752000,"data_stored":15624892,"data_compressed":268173,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280536,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42180,"kb_used_data":10992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94329660,"statfs":{"total":96636764160,"available":96593571840,"internally_reserved":0,"allocated":11255808,"data_stored":13293871,"data_compressed":274262,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280535,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47040,"kb_used_data":15852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94324800,"statfs":{"total":96636764160,"available":96588595200,"internally_reserved":0,"allocated":16232448,"data_stored":22128962,"data_compressed":274262,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44016,"kb_used_data":12828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327824,"statfs":{"total":96636764160,"available":96591691776,"internally_reserved":0,"allocated":13135872,"data_stored":16798264,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280536,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50948,"kb_used_data":19760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94320892,"statfs":{"total":96636764160,"available":96584593408,"internally_reserved":0,"allocated":20234240,"data_stored":26575960,"data_compressed":303757,"data_compressed_allocated":413696,"data_compressed_original":1257472,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44700,"kb_used_data":13512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94327140,"statfs":{"total":96636764160,"available":96590991360,"internally_reserved":0,"allocated":13836288,"data_stored":17908280,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280536,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37464,"kb_used_data":10436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94334376,"statfs":{"total":96636764160,"available":96598401024,"internally_reserved":0,"allocated":10686464,"data_stored":13292088,"data_compressed":271226,"data_compressed_allocated":372736,"data_compressed_original":745472,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280535,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45988,"kb_used_data":14800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325852,"statfs":{"total":96636764160,"available":96589672448,"internally_reserved":0,"allocated":15155200,"data_stored":21043881,"data_compressed":268173,"data_compressed_allocated":368640,"data_compressed_original":737280,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280536,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45352,"kb_used_data":14164,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94326488,"statfs":{"total":96636764160,"available":96590323712,"internally_reserved":0,"allocated":14503936,"data_stored":18766127,"data_compressed":274262,"data_compressed_allocated":376832,"data_compressed_original":753664,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":0,"seq":429496729605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":18853888,"data_stored":25047040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":15015936,"data_stored":21540864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9441280,"data_stored":14635008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12517376,"data_stored":16908288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":9367552,"data_stored":12292096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":9924608,"data_stored":12283904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":13848576,"data_stored":20054016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":14897152,"data_stored":21118976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":11816960,"data_stored":15798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":13172736,"data_stored":17756160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7139328,"data_stored":8015872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1248,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8900608,"data_stored":12447744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":12349440,"data_stored":18518016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":11288576,"data_stored":17813504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":17993728,"data_stored":23339008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":16297984,"data_stored":22544384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:15:51.254 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-16T02:15:51.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-16T02:15:51.325 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:51.319+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:51.331 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.331 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.345 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-16T02:15:51.425 INFO:tasks.ceph.osd.5.smithi118.stderr:2024-09-16T02:15:51.423+0000 7f4528a70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26348) UID: 0 2024-09-16T02:15:51.465 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.465 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.481 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-16T02:15:51.526 INFO:tasks.ceph.osd.4.smithi003.stderr:2024-09-16T02:15:51.526+0000 7f589ffc1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26327) UID: 0 2024-09-16T02:15:51.614 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.614 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.627 INFO:tasks.ceph.osd.11.smithi195.stderr:2024-09-16T02:15:51.621+0000 7f3acf318640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26579) UID: 0 2024-09-16T02:15:51.629 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-16T02:15:51.728 INFO:tasks.ceph.osd.3.smithi195.stderr:2024-09-16T02:15:51.721+0000 7f8fee2f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26577) UID: 0 2024-09-16T02:15:51.798 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.798 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.812 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-16T02:15:51.829 INFO:tasks.ceph.osd.9.smithi118.stderr:2024-09-16T02:15:51.827+0000 7f379557c640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26350) UID: 0 2024-09-16T02:15:51.929 INFO:tasks.ceph.osd.13.smithi118.stderr:2024-09-16T02:15:51.923+0000 7f652659e640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26352) UID: 0 2024-09-16T02:15:51.979 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:51.979 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:51.994 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-16T02:15:52.030 INFO:tasks.ceph.osd.1.smithi118.stderr:2024-09-16T02:15:52.023+0000 7f51b14ce640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26346) UID: 0 2024-09-16T02:15:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:52.067 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' removed 2024-09-16T02:15:52.100 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-16T02:15:52.111 INFO:tasks.thrashosds:joining thrashosds 2024-09-16T02:15:52.123 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.123 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.137 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-16T02:15:52.259 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.259 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.273 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-16T02:15:52.390 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.390 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.405 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-16T02:15:52.522 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.522 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.536 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-16T02:15:52.656 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.656 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-16T02:15:52.800 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.801 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.816 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-16T02:15:52.941 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:52.941 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:52.955 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-16T02:15:53.078 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:53.078 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:53.094 DEBUG:teuthology.orchestra.run.smithi195:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-16T02:15:53.217 INFO:teuthology.orchestra.run.smithi195.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:53.217 INFO:teuthology.orchestra.run.smithi195.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:53.231 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-16T02:15:53.364 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:53.364 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:53.380 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-16T02:15:53.503 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:53.503 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:53.517 DEBUG:teuthology.orchestra.run.smithi177:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-16T02:15:53.634 INFO:teuthology.orchestra.run.smithi177.stdout:ERROR: (22) Invalid argument 2024-09-16T02:15:53.634 INFO:teuthology.orchestra.run.smithi177.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-16T02:15:54.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-16T02:15:55.089 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-09-16T02:15:55.090 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-16T02:15:55.090 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-16T02:15:55.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 10 2024-09-16T02:15:55.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-16T02:15:55.419 INFO:tasks.ceph.mon.a.smithi003.stderr:2024-09-16T02:15:55.418+0000 7f5f54a8b640 -1 mon.a@0(leader).osd e117 definitely_dead 0 2024-09-16T02:15:56.098 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.10. 2024-09-16T02:15:56.098 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-09-16T02:15:56.126 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-16T02:15:56.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-16T02:15:56.321 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-09-16T02:15:56.321 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:15:56.339 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-09-16T02:15:56.339 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:15:56.358 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-09-16T02:15:56.358 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-16T02:15:56.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-16T02:15:57.100 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-09-16T02:15:57.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-16T02:15:58.110 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-16T02:15:58.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-16T02:15:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-09-16T02:15:59.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-16T02:16:00.134 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-16T02:16:00.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-16T02:16:01.161 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.8 to 1 (10000) 2024-09-16T02:16:01.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-16T02:16:02.163 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-16T02:16:02.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-16T02:16:03.181 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.12 to 1 (10000) 2024-09-16T02:16:03.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-16T02:16:04.191 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-16T02:16:04.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-16T02:16:05.202 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-09-16T02:16:05.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-16T02:16:06.213 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-16T02:16:06.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-16T02:16:07.222 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-09-16T02:16:07.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-16T02:16:08.238 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-16T02:16:08.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-16T02:16:09.241 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.9 to 1 (10000) 2024-09-16T02:16:09.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-16T02:16:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-16T02:16:10.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-16T02:16:11.259 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.13 to 1 (10000) 2024-09-16T02:16:11.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-16T02:16:12.277 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-16T02:16:12.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-16T02:16:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-09-16T02:16:13.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-16T02:16:14.291 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-16T02:16:14.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-16T02:16:15.308 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-09-16T02:16:15.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-16T02:16:16.311 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-16T02:16:16.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-16T02:16:17.323 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.14 to 1 (10000) 2024-09-16T02:16:17.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-16T02:16:18.336 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-16T02:16:18.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-16T02:16:19.348 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-09-16T02:16:19.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-16T02:16:20.368 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-16T02:16:20.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-16T02:16:21.370 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-09-16T02:16:21.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-16T02:16:22.388 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-16T02:16:22.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-16T02:16:23.391 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.11 to 1 (10000) 2024-09-16T02:16:23.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-16T02:16:24.411 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-16T02:16:24.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-16T02:16:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.15 to 1 (10000) 2024-09-16T02:16:25.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-16T02:16:26.430 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-16T02:16:26.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-16T02:16:27.433 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.10 to 1 (10000) 2024-09-16T02:16:27.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-16T02:16:28.443 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-16T02:16:28.470 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-16T02:16:28.471 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-16T02:16:28.471 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-16T02:16:28.471 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-16T02:16:28.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-16T02:16:28.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:16:28.799 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:28.799 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:16:28.440002+0000","last_up_change":"2024-09-16T02:15:28.919687+0000","last_in_change":"2024-09-16T02:15:56.089801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":81,"up_from":100,"up_thru":32,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6809","nonce":3610577698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6811","nonce":3610577698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6815","nonce":3610577698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6813","nonce":3610577698}]},"public_addr":"172.21.15.177:6809/3610577698","cluster_addr":"172.21.15.177:6811/3610577698","heartbeat_back_addr":"172.21.15.177:6815/3610577698","heartbeat_front_addr":"172.21.15.177:6813/3610577698","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:10.406807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:20.891965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:25.986137+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:16.290057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:10.699837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:09.304914+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:20.841771+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:22.263679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:18.765693+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:26.923353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:28.101178+0000","dead_epoch":82},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.748260+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:19.298920+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.709317+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.987870+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.522681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:16:28.819 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-16T02:16:28.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:16:29.137 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:29.137 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":150,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:16:28.440002+0000","last_up_change":"2024-09-16T02:15:28.919687+0000","last_in_change":"2024-09-16T02:15:56.089801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":81,"up_from":100,"up_thru":32,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6809","nonce":3610577698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6811","nonce":3610577698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6815","nonce":3610577698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6813","nonce":3610577698}]},"public_addr":"172.21.15.177:6809/3610577698","cluster_addr":"172.21.15.177:6811/3610577698","heartbeat_back_addr":"172.21.15.177:6815/3610577698","heartbeat_front_addr":"172.21.15.177:6813/3610577698","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:10.406807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:20.891965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:25.986137+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:16.290057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:10.699837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:09.304914+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:20.841771+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:22.263679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:18.765693+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:26.923353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:28.101178+0000","dead_epoch":82},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.748260+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:19.298920+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.709317+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.987870+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.522681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:16:29.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-16T02:16:29.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-16T02:16:29.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-16T02:16:29.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-16T02:16:29.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-16T02:16:29.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-16T02:16:29.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-16T02:16:29.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-16T02:16:29.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-16T02:16:29.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-16T02:16:29.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-16T02:16:29.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-16T02:16:29.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-16T02:16:29.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-16T02:16:29.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-16T02:16:29.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-16T02:16:29.580 INFO:teuthology.orchestra.run.smithi003.stdout:90194313250 2024-09-16T02:16:29.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T02:16:29.601 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:29.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:16:29.603 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-16T02:16:29.632 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-16T02:16:29.639 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-16T02:16:29.663 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-16T02:16:29.685 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-16T02:16:29.692 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-16T02:16:29.701 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:29.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-16T02:16:29.718 INFO:teuthology.orchestra.run.smithi003.stdout:429496729614 2024-09-16T02:16:29.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-16T02:16:29.752 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-16T02:16:29.757 INFO:teuthology.orchestra.run.smithi003.stdout:90194313248 2024-09-16T02:16:29.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-16T02:16:29.762 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-16T02:16:29.767 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:29.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-16T02:16:29.773 INFO:teuthology.orchestra.run.smithi003.stdout:94489280543 2024-09-16T02:16:29.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-16T02:16:29.780 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:29.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-16T02:16:30.389 INFO:teuthology.orchestra.run.smithi003.stdout:90194313248 2024-09-16T02:16:30.406 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.444 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313248 for osd.14 2024-09-16T02:16:30.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280544 for osd.1 2024-09-16T02:16:30.620 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.669 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.0 2024-09-16T02:16:30.669 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.679 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.3 2024-09-16T02:16:30.720 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.758 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.762 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:30.767 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.768 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.768 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.797 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.11 2024-09-16T02:16:30.797 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.813 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.9 2024-09-16T02:16:30.813 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.813 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.4 2024-09-16T02:16:30.817 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.818 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.6 2024-09-16T02:16:30.818 INFO:teuthology.orchestra.run.smithi003.stdout:90194313248 2024-09-16T02:16:30.818 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.827 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.13 2024-09-16T02:16:30.827 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.5 2024-09-16T02:16:30.845 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313248 got 90194313248 for osd.15 2024-09-16T02:16:30.847 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.847 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.851 INFO:teuthology.orchestra.run.smithi003.stdout:94489280544 2024-09-16T02:16:30.864 INFO:teuthology.orchestra.run.smithi003.stdout:429496729614 2024-09-16T02:16:30.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.2 2024-09-16T02:16:30.871 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.875 INFO:teuthology.orchestra.run.smithi003.stdout:94489280543 2024-09-16T02:16:30.876 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280544 got 94489280544 for osd.12 2024-09-16T02:16:30.876 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.884 INFO:tasks.ceph.ceph_manager.ceph:need seq 429496729614 got 429496729614 for osd.10 2024-09-16T02:16:30.885 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:30.885 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280543 got 94489280543 for osd.7 2024-09-16T02:16:30.896 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:30.903 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.8 2024-09-16T02:16:30.903 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:31.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-16T02:16:31.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-16T02:16:31.772 INFO:teuthology.orchestra.run.smithi003.stdout:90194313250 2024-09-16T02:16:31.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313250 got 90194313250 for osd.14 2024-09-16T02:16:31.790 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:31.799 INFO:teuthology.orchestra.run.smithi003.stdout:94489280545 2024-09-16T02:16:31.816 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280545 got 94489280545 for osd.1 2024-09-16T02:16:31.817 DEBUG:teuthology.parallel:result is None 2024-09-16T02:16:31.817 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-16T02:16:31.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:32.153 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:32.153 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:32.170 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:16:32.036546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":527484,"kb_used_data":27856,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421956,"statfs":{"total":1546188226560,"available":1545648082944,"internally_reserved":0,"allocated":28524544,"data_stored":22707902,"data_compressed":6166350,"data_compressed_allocated":8454144,"data_compressed_original":17768448,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605214"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":392,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447030+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:16:28.447030+0000","last_peered":"2024-09-16T02:16:28.447030+0000","last_clean":"2024-09-16T02:16:28.447030+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:28.447030+0000","last_undegraded":"2024-09-16T02:16:28.447030+0000","last_fullsized":"2024-09-16T02:16:28.447030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":263,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.446400+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:16:28.446400+0000","last_peered":"2024-09-16T02:16:28.446400+0000","last_clean":"2024-09-16T02:16:28.446400+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:28.446400+0000","last_undegraded":"2024-09-16T02:16:28.446400+0000","last_fullsized":"2024-09-16T02:16:28.446400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537450+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:16:29.537450+0000","last_peered":"2024-09-16T02:16:29.537450+0000","last_clean":"2024-09-16T02:16:29.537450+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:29.537450+0000","last_undegraded":"2024-09-16T02:16:29.537450+0000","last_fullsized":"2024-09-16T02:16:29.537450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447075+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:28.447075+0000","last_peered":"2024-09-16T02:16:28.447075+0000","last_clean":"2024-09-16T02:16:28.447075+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:28.447075+0000","last_undegraded":"2024-09-16T02:16:28.447075+0000","last_fullsized":"2024-09-16T02:16:28.447075+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":268,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537514+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:29.537514+0000","last_peered":"2024-09-16T02:16:29.537514+0000","last_clean":"2024-09-16T02:16:29.537514+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:29.537514+0000","last_undegraded":"2024-09-16T02:16:29.537514+0000","last_fullsized":"2024-09-16T02:16:29.537514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.546398+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:16:29.546398+0000","last_peered":"2024-09-16T02:16:29.546398+0000","last_clean":"2024-09-16T02:16:29.546398+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:29.546398+0000","last_undegraded":"2024-09-16T02:16:29.546398+0000","last_fullsized":"2024-09-16T02:16:29.546398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.536931+0000","last_change":"2024-09-16T02:16:23.547758+0000","last_active":"2024-09-16T02:16:29.536931+0000","last_peered":"2024-09-16T02:16:29.536931+0000","last_clean":"2024-09-16T02:16:29.536931+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:29.536931+0000","last_undegraded":"2024-09-16T02:16:29.536931+0000","last_fullsized":"2024-09-16T02:16:29.536931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:23.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:23.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.935295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.470772+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:28.470772+0000","last_peered":"2024-09-16T02:16:28.470772+0000","last_clean":"2024-09-16T02:16:28.470772+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:28.470772+0000","last_undegraded":"2024-09-16T02:16:28.470772+0000","last_fullsized":"2024-09-16T02:16:28.470772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537410+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:29.537410+0000","last_peered":"2024-09-16T02:16:29.537410+0000","last_clean":"2024-09-16T02:16:29.537410+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:29.537410+0000","last_undegraded":"2024-09-16T02:16:29.537410+0000","last_fullsized":"2024-09-16T02:16:29.537410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1839104,"data_stored":1884443,"data_compressed":414610,"data_compressed_allocated":565248,"data_compressed_original":1560576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32932,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1874885,"data_compressed":411616,"data_compressed_allocated":561152,"data_compressed_original":1552384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28764,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32936,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29328,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:32.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:32.477 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:32.478 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:32.495 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:16:32.036546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":527484,"kb_used_data":27856,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421956,"statfs":{"total":1546188226560,"available":1545648082944,"internally_reserved":0,"allocated":28524544,"data_stored":22707902,"data_compressed":6166350,"data_compressed_allocated":8454144,"data_compressed_original":17768448,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605214"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":392,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447030+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:16:28.447030+0000","last_peered":"2024-09-16T02:16:28.447030+0000","last_clean":"2024-09-16T02:16:28.447030+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:28.447030+0000","last_undegraded":"2024-09-16T02:16:28.447030+0000","last_fullsized":"2024-09-16T02:16:28.447030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":263,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.446400+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:16:28.446400+0000","last_peered":"2024-09-16T02:16:28.446400+0000","last_clean":"2024-09-16T02:16:28.446400+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:28.446400+0000","last_undegraded":"2024-09-16T02:16:28.446400+0000","last_fullsized":"2024-09-16T02:16:28.446400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537450+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:16:29.537450+0000","last_peered":"2024-09-16T02:16:29.537450+0000","last_clean":"2024-09-16T02:16:29.537450+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:29.537450+0000","last_undegraded":"2024-09-16T02:16:29.537450+0000","last_fullsized":"2024-09-16T02:16:29.537450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447075+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:28.447075+0000","last_peered":"2024-09-16T02:16:28.447075+0000","last_clean":"2024-09-16T02:16:28.447075+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:28.447075+0000","last_undegraded":"2024-09-16T02:16:28.447075+0000","last_fullsized":"2024-09-16T02:16:28.447075+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":268,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537514+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:29.537514+0000","last_peered":"2024-09-16T02:16:29.537514+0000","last_clean":"2024-09-16T02:16:29.537514+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:29.537514+0000","last_undegraded":"2024-09-16T02:16:29.537514+0000","last_fullsized":"2024-09-16T02:16:29.537514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.546398+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:16:29.546398+0000","last_peered":"2024-09-16T02:16:29.546398+0000","last_clean":"2024-09-16T02:16:29.546398+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:29.546398+0000","last_undegraded":"2024-09-16T02:16:29.546398+0000","last_fullsized":"2024-09-16T02:16:29.546398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.536931+0000","last_change":"2024-09-16T02:16:23.547758+0000","last_active":"2024-09-16T02:16:29.536931+0000","last_peered":"2024-09-16T02:16:29.536931+0000","last_clean":"2024-09-16T02:16:29.536931+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:29.536931+0000","last_undegraded":"2024-09-16T02:16:29.536931+0000","last_fullsized":"2024-09-16T02:16:29.536931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:23.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:23.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.935295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.470772+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:28.470772+0000","last_peered":"2024-09-16T02:16:28.470772+0000","last_clean":"2024-09-16T02:16:28.470772+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:28.470772+0000","last_undegraded":"2024-09-16T02:16:28.470772+0000","last_fullsized":"2024-09-16T02:16:28.470772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537410+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:29.537410+0000","last_peered":"2024-09-16T02:16:29.537410+0000","last_clean":"2024-09-16T02:16:29.537410+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:29.537410+0000","last_undegraded":"2024-09-16T02:16:29.537410+0000","last_fullsized":"2024-09-16T02:16:29.537410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1839104,"data_stored":1884443,"data_compressed":414610,"data_compressed_allocated":565248,"data_compressed_original":1560576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32932,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1874885,"data_compressed":411616,"data_compressed_allocated":561152,"data_compressed_original":1552384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28764,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32936,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29328,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:32.496 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-16T02:16:32.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-16T02:16:32.827 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T02:16:32.835 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-16T02:16:32.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:33.143 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:33.144 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:33.160 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:16:32.036546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":527484,"kb_used_data":27856,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421956,"statfs":{"total":1546188226560,"available":1545648082944,"internally_reserved":0,"allocated":28524544,"data_stored":22707902,"data_compressed":6166350,"data_compressed_allocated":8454144,"data_compressed_original":17768448,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605214"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":392,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447030+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:16:28.447030+0000","last_peered":"2024-09-16T02:16:28.447030+0000","last_clean":"2024-09-16T02:16:28.447030+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:28.447030+0000","last_undegraded":"2024-09-16T02:16:28.447030+0000","last_fullsized":"2024-09-16T02:16:28.447030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":263,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.446400+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:16:28.446400+0000","last_peered":"2024-09-16T02:16:28.446400+0000","last_clean":"2024-09-16T02:16:28.446400+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:28.446400+0000","last_undegraded":"2024-09-16T02:16:28.446400+0000","last_fullsized":"2024-09-16T02:16:28.446400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537450+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:16:29.537450+0000","last_peered":"2024-09-16T02:16:29.537450+0000","last_clean":"2024-09-16T02:16:29.537450+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:29.537450+0000","last_undegraded":"2024-09-16T02:16:29.537450+0000","last_fullsized":"2024-09-16T02:16:29.537450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447075+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:28.447075+0000","last_peered":"2024-09-16T02:16:28.447075+0000","last_clean":"2024-09-16T02:16:28.447075+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:28.447075+0000","last_undegraded":"2024-09-16T02:16:28.447075+0000","last_fullsized":"2024-09-16T02:16:28.447075+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":268,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537514+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:29.537514+0000","last_peered":"2024-09-16T02:16:29.537514+0000","last_clean":"2024-09-16T02:16:29.537514+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:29.537514+0000","last_undegraded":"2024-09-16T02:16:29.537514+0000","last_fullsized":"2024-09-16T02:16:29.537514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.546398+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:16:29.546398+0000","last_peered":"2024-09-16T02:16:29.546398+0000","last_clean":"2024-09-16T02:16:29.546398+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:29.546398+0000","last_undegraded":"2024-09-16T02:16:29.546398+0000","last_fullsized":"2024-09-16T02:16:29.546398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.536931+0000","last_change":"2024-09-16T02:16:23.547758+0000","last_active":"2024-09-16T02:16:29.536931+0000","last_peered":"2024-09-16T02:16:29.536931+0000","last_clean":"2024-09-16T02:16:29.536931+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:29.536931+0000","last_undegraded":"2024-09-16T02:16:29.536931+0000","last_fullsized":"2024-09-16T02:16:29.536931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:23.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:23.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.935295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.470772+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:28.470772+0000","last_peered":"2024-09-16T02:16:28.470772+0000","last_clean":"2024-09-16T02:16:28.470772+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:28.470772+0000","last_undegraded":"2024-09-16T02:16:28.470772+0000","last_fullsized":"2024-09-16T02:16:28.470772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537410+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:29.537410+0000","last_peered":"2024-09-16T02:16:29.537410+0000","last_clean":"2024-09-16T02:16:29.537410+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:29.537410+0000","last_undegraded":"2024-09-16T02:16:29.537410+0000","last_fullsized":"2024-09-16T02:16:29.537410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1839104,"data_stored":1884443,"data_compressed":414610,"data_compressed_allocated":565248,"data_compressed_original":1560576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32932,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1874885,"data_compressed":411616,"data_compressed_allocated":561152,"data_compressed_original":1552384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28764,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32936,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29328,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:33.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:33.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:33.458 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:33.474 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:16:32.036546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":527484,"kb_used_data":27856,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421956,"statfs":{"total":1546188226560,"available":1545648082944,"internally_reserved":0,"allocated":28524544,"data_stored":22707902,"data_compressed":6166350,"data_compressed_allocated":8454144,"data_compressed_original":17768448,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605214"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":392,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447030+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:16:28.447030+0000","last_peered":"2024-09-16T02:16:28.447030+0000","last_clean":"2024-09-16T02:16:28.447030+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:28.447030+0000","last_undegraded":"2024-09-16T02:16:28.447030+0000","last_fullsized":"2024-09-16T02:16:28.447030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":263,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.446400+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:16:28.446400+0000","last_peered":"2024-09-16T02:16:28.446400+0000","last_clean":"2024-09-16T02:16:28.446400+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:28.446400+0000","last_undegraded":"2024-09-16T02:16:28.446400+0000","last_fullsized":"2024-09-16T02:16:28.446400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537450+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:16:29.537450+0000","last_peered":"2024-09-16T02:16:29.537450+0000","last_clean":"2024-09-16T02:16:29.537450+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:29.537450+0000","last_undegraded":"2024-09-16T02:16:29.537450+0000","last_fullsized":"2024-09-16T02:16:29.537450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447075+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:28.447075+0000","last_peered":"2024-09-16T02:16:28.447075+0000","last_clean":"2024-09-16T02:16:28.447075+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:28.447075+0000","last_undegraded":"2024-09-16T02:16:28.447075+0000","last_fullsized":"2024-09-16T02:16:28.447075+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":268,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537514+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:29.537514+0000","last_peered":"2024-09-16T02:16:29.537514+0000","last_clean":"2024-09-16T02:16:29.537514+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:29.537514+0000","last_undegraded":"2024-09-16T02:16:29.537514+0000","last_fullsized":"2024-09-16T02:16:29.537514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.546398+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:16:29.546398+0000","last_peered":"2024-09-16T02:16:29.546398+0000","last_clean":"2024-09-16T02:16:29.546398+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:29.546398+0000","last_undegraded":"2024-09-16T02:16:29.546398+0000","last_fullsized":"2024-09-16T02:16:29.546398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.536931+0000","last_change":"2024-09-16T02:16:23.547758+0000","last_active":"2024-09-16T02:16:29.536931+0000","last_peered":"2024-09-16T02:16:29.536931+0000","last_clean":"2024-09-16T02:16:29.536931+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:29.536931+0000","last_undegraded":"2024-09-16T02:16:29.536931+0000","last_fullsized":"2024-09-16T02:16:29.536931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:23.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:23.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.935295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.470772+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:28.470772+0000","last_peered":"2024-09-16T02:16:28.470772+0000","last_clean":"2024-09-16T02:16:28.470772+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:28.470772+0000","last_undegraded":"2024-09-16T02:16:28.470772+0000","last_fullsized":"2024-09-16T02:16:28.470772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537410+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:29.537410+0000","last_peered":"2024-09-16T02:16:29.537410+0000","last_clean":"2024-09-16T02:16:29.537410+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:29.537410+0000","last_undegraded":"2024-09-16T02:16:29.537410+0000","last_fullsized":"2024-09-16T02:16:29.537410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1839104,"data_stored":1884443,"data_compressed":414610,"data_compressed_allocated":565248,"data_compressed_original":1560576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32932,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1874885,"data_compressed":411616,"data_compressed_allocated":561152,"data_compressed_original":1552384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28764,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32936,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29328,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:33.475 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-16T02:16:33.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:33.821 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:33.839 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-16T02:16:32.036546+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":527484,"kb_used_data":27856,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421956,"statfs":{"total":1546188226560,"available":1545648082944,"internally_reserved":0,"allocated":28524544,"data_stored":22707902,"data_compressed":6166350,"data_compressed_allocated":8454144,"data_compressed_original":17768448,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.605214"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":392,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447030+0000","last_change":"2024-09-16T02:14:06.360685+0000","last_active":"2024-09-16T02:16:28.447030+0000","last_peered":"2024-09-16T02:16:28.447030+0000","last_clean":"2024-09-16T02:16:28.447030+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:28.447030+0000","last_undegraded":"2024-09-16T02:16:28.447030+0000","last_fullsized":"2024-09-16T02:16:28.447030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:14:05.038727+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:32.120515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":263,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.446400+0000","last_change":"2024-09-16T02:14:10.306458+0000","last_active":"2024-09-16T02:16:28.446400+0000","last_peered":"2024-09-16T02:16:28.446400+0000","last_clean":"2024-09-16T02:16:28.446400+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:28.446400+0000","last_undegraded":"2024-09-16T02:16:28.446400+0000","last_fullsized":"2024-09-16T02:16:28.446400+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:34.757242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":258,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537450+0000","last_change":"2024-09-16T02:14:10.179566+0000","last_active":"2024-09-16T02:16:29.537450+0000","last_peered":"2024-09-16T02:16:29.537450+0000","last_clean":"2024-09-16T02:16:29.537450+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:29.537450+0000","last_undegraded":"2024-09-16T02:16:29.537450+0000","last_fullsized":"2024-09-16T02:16:29.537450+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":264,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.447075+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:28.447075+0000","last_peered":"2024-09-16T02:16:28.447075+0000","last_clean":"2024-09-16T02:16:28.447075+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:28.447075+0000","last_undegraded":"2024-09-16T02:16:28.447075+0000","last_fullsized":"2024-09-16T02:16:28.447075+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":268,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537514+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:29.537514+0000","last_peered":"2024-09-16T02:16:29.537514+0000","last_clean":"2024-09-16T02:16:29.537514+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:29.537514+0000","last_undegraded":"2024-09-16T02:16:29.537514+0000","last_fullsized":"2024-09-16T02:16:29.537514+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:36.208020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.546398+0000","last_change":"2024-09-16T02:14:55.927932+0000","last_active":"2024-09-16T02:16:29.546398+0000","last_peered":"2024-09-16T02:16:29.546398+0000","last_clean":"2024-09-16T02:16:29.546398+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:29.546398+0000","last_undegraded":"2024-09-16T02:16:29.546398+0000","last_fullsized":"2024-09-16T02:16:29.546398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:14:07.079827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:31.459781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.536931+0000","last_change":"2024-09-16T02:16:23.547758+0000","last_active":"2024-09-16T02:16:29.536931+0000","last_peered":"2024-09-16T02:16:29.536931+0000","last_clean":"2024-09-16T02:16:29.536931+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:29.536931+0000","last_undegraded":"2024-09-16T02:16:29.536931+0000","last_fullsized":"2024-09-16T02:16:29.536931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:23.547680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:23.547680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.935295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":269,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:28.470772+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:28.470772+0000","last_peered":"2024-09-16T02:16:28.470772+0000","last_clean":"2024-09-16T02:16:28.470772+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:28.470772+0000","last_undegraded":"2024-09-16T02:16:28.470772+0000","last_fullsized":"2024-09-16T02:16:28.470772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":267,"reported_epoch":150,"state":"active+clean","last_fresh":"2024-09-16T02:16:29.537410+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:29.537410+0000","last_peered":"2024-09-16T02:16:29.537410+0000","last_clean":"2024-09-16T02:16:29.537410+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:29.537410+0000","last_undegraded":"2024-09-16T02:16:29.537410+0000","last_fullsized":"2024-09-16T02:16:29.537410+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37148,"kb_used_data":1796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334692,"statfs":{"total":96636764160,"available":96598724608,"internally_reserved":0,"allocated":1839104,"data_stored":1884443,"data_compressed":414610,"data_compressed_allocated":565248,"data_compressed_original":1560576,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32932,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338908,"statfs":{"total":96636764160,"available":96603041792,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280546,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37084,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334756,"statfs":{"total":96636764160,"available":96598790144,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313250,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280546,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32916,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338924,"statfs":{"total":96636764160,"available":96603058176,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280545,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280544,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1781760,"data_stored":1355534,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280545,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1874885,"data_compressed":411616,"data_compressed_allocated":561152,"data_compressed_original":1552384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28764,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343076,"statfs":{"total":96636764160,"available":96607309824,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280545,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32924,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338916,"statfs":{"total":96636764160,"available":96603049984,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280545,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32936,"kb_used_data":1736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338904,"statfs":{"total":96636764160,"available":96603037696,"internally_reserved":0,"allocated":1777664,"data_stored":1355515,"data_compressed":382079,"data_compressed_allocated":524288,"data_compressed_original":1048576,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729614,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29328,"kb_used_data":1724,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342512,"statfs":{"total":96636764160,"available":96606732288,"internally_reserved":0,"allocated":1765376,"data_stored":1345957,"data_compressed":379085,"data_compressed_allocated":520192,"data_compressed_original":1040384,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:33.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-16T02:16:34.146 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:34.146 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":151,"fsid":"babbcdda-abe3-4b2f-82b4-da47bc408d34","created":"2024-09-16T02:13:51.176653+0000","modified":"2024-09-16T02:16:31.467978+0000","last_up_change":"2024-09-16T02:15:28.919687+0000","last_in_change":"2024-09-16T02:15:56.089801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T02:14:04.065882+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T02:14:06.221379+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.0199999809265137,"score_stable":4.0199999809265137,"optimal_score":0.56000000238418579,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d00dcd0d-0b5f-48c9-8d98-adaaabf5283a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6809","nonce":3190094522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6811","nonce":3190094522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6815","nonce":3190094522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3190094522},{"type":"v1","addr":"172.21.15.3:6813","nonce":3190094522}]},"public_addr":"172.21.15.3:6809/3190094522","cluster_addr":"172.21.15.3:6811/3190094522","heartbeat_back_addr":"172.21.15.3:6815/3190094522","heartbeat_front_addr":"172.21.15.3:6813/3190094522","state":["exists","up"]},{"osd":1,"uuid":"56b3dd3b-fe4d-47ae-b56e-34ab7ff7423c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6817","nonce":1191817370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6819","nonce":1191817370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6823","nonce":1191817370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":1191817370},{"type":"v1","addr":"172.21.15.118:6821","nonce":1191817370}]},"public_addr":"172.21.15.118:6817/1191817370","cluster_addr":"172.21.15.118:6819/1191817370","heartbeat_back_addr":"172.21.15.118:6823/1191817370","heartbeat_front_addr":"172.21.15.118:6821/1191817370","state":["exists","up"]},{"osd":2,"uuid":"6bfe62d5-9c5b-44df-bc4f-cf312837c932","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6817","nonce":3704126567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6819","nonce":3704126567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6823","nonce":3704126567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3704126567},{"type":"v1","addr":"172.21.15.177:6821","nonce":3704126567}]},"public_addr":"172.21.15.177:6817/3704126567","cluster_addr":"172.21.15.177:6819/3704126567","heartbeat_back_addr":"172.21.15.177:6823/3704126567","heartbeat_front_addr":"172.21.15.177:6821/3704126567","state":["exists","up"]},{"osd":3,"uuid":"fe2b6f49-91ab-4692-ab9c-4b3bda3bf429","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6826","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6827","nonce":3376854549}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6828","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6829","nonce":3376854549}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6832","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6833","nonce":3376854549}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6830","nonce":3376854549},{"type":"v1","addr":"172.21.15.195:6831","nonce":3376854549}]},"public_addr":"172.21.15.195:6827/3376854549","cluster_addr":"172.21.15.195:6829/3376854549","heartbeat_back_addr":"172.21.15.195:6833/3376854549","heartbeat_front_addr":"172.21.15.195:6831/3376854549","state":["exists","up"]},{"osd":4,"uuid":"399375fc-ce28-40e2-89b1-e0fc2c1f44ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6825","nonce":1858329864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6827","nonce":1858329864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6831","nonce":1858329864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":1858329864},{"type":"v1","addr":"172.21.15.3:6829","nonce":1858329864}]},"public_addr":"172.21.15.3:6825/1858329864","cluster_addr":"172.21.15.3:6827/1858329864","heartbeat_back_addr":"172.21.15.3:6831/1858329864","heartbeat_front_addr":"172.21.15.3:6829/1858329864","state":["exists","up"]},{"osd":5,"uuid":"3709bae2-c1ac-45f2-b1a3-6271ca5d6aa3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6801","nonce":2650650539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6803","nonce":2650650539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6807","nonce":2650650539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":2650650539},{"type":"v1","addr":"172.21.15.118:6805","nonce":2650650539}]},"public_addr":"172.21.15.118:6801/2650650539","cluster_addr":"172.21.15.118:6803/2650650539","heartbeat_back_addr":"172.21.15.118:6807/2650650539","heartbeat_front_addr":"172.21.15.118:6805/2650650539","state":["exists","up"]},{"osd":6,"uuid":"ac6efa66-27ed-494c-956d-22b90be7a8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6825","nonce":47176522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6827","nonce":47176522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6831","nonce":47176522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":47176522},{"type":"v1","addr":"172.21.15.177:6829","nonce":47176522}]},"public_addr":"172.21.15.177:6825/47176522","cluster_addr":"172.21.15.177:6827/47176522","heartbeat_back_addr":"172.21.15.177:6831/47176522","heartbeat_front_addr":"172.21.15.177:6829/47176522","state":["exists","up"]},{"osd":7,"uuid":"fb2a81f7-b4f5-4c4c-aff8-801e8b73d192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6810","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6811","nonce":3756385243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6812","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6813","nonce":3756385243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6816","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6817","nonce":3756385243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6814","nonce":3756385243},{"type":"v1","addr":"172.21.15.195:6815","nonce":3756385243}]},"public_addr":"172.21.15.195:6811/3756385243","cluster_addr":"172.21.15.195:6813/3756385243","heartbeat_back_addr":"172.21.15.195:6817/3756385243","heartbeat_front_addr":"172.21.15.195:6815/3756385243","state":["exists","up"]},{"osd":8,"uuid":"01aca5fb-932f-4ecb-a9ab-c716a5247373","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6817","nonce":3976393740}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6819","nonce":3976393740}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6823","nonce":3976393740}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":3976393740},{"type":"v1","addr":"172.21.15.3:6821","nonce":3976393740}]},"public_addr":"172.21.15.3:6817/3976393740","cluster_addr":"172.21.15.3:6819/3976393740","heartbeat_back_addr":"172.21.15.3:6823/3976393740","heartbeat_front_addr":"172.21.15.3:6821/3976393740","state":["exists","up"]},{"osd":9,"uuid":"d52efa47-5645-4018-b495-ed116da5f7a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6825","nonce":2321182247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6827","nonce":2321182247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6831","nonce":2321182247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":2321182247},{"type":"v1","addr":"172.21.15.118:6829","nonce":2321182247}]},"public_addr":"172.21.15.118:6825/2321182247","cluster_addr":"172.21.15.118:6827/2321182247","heartbeat_back_addr":"172.21.15.118:6831/2321182247","heartbeat_front_addr":"172.21.15.118:6829/2321182247","state":["exists","up"]},{"osd":10,"uuid":"99005d53-e62d-4c20-8d3c-541fd20cbeb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":81,"up_from":100,"up_thru":32,"down_at":82,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6809","nonce":3610577698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6811","nonce":3610577698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6815","nonce":3610577698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":3610577698},{"type":"v1","addr":"172.21.15.177:6813","nonce":3610577698}]},"public_addr":"172.21.15.177:6809/3610577698","cluster_addr":"172.21.15.177:6811/3610577698","heartbeat_back_addr":"172.21.15.177:6815/3610577698","heartbeat_front_addr":"172.21.15.177:6813/3610577698","state":["exists","up"]},{"osd":11,"uuid":"f460f1c7-0172-4bd9-875f-b34aff85804f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6818","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6819","nonce":1553304829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6820","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6821","nonce":1553304829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6824","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6825","nonce":1553304829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6822","nonce":1553304829},{"type":"v1","addr":"172.21.15.195:6823","nonce":1553304829}]},"public_addr":"172.21.15.195:6819/1553304829","cluster_addr":"172.21.15.195:6821/1553304829","heartbeat_back_addr":"172.21.15.195:6825/1553304829","heartbeat_front_addr":"172.21.15.195:6823/1553304829","state":["exists","up"]},{"osd":12,"uuid":"7bd5fea1-6e94-478a-8a8f-87459b4030a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6801","nonce":3221525577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6803","nonce":3221525577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6807","nonce":3221525577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":3221525577},{"type":"v1","addr":"172.21.15.3:6805","nonce":3221525577}]},"public_addr":"172.21.15.3:6801/3221525577","cluster_addr":"172.21.15.3:6803/3221525577","heartbeat_back_addr":"172.21.15.3:6807/3221525577","heartbeat_front_addr":"172.21.15.3:6805/3221525577","state":["exists","up"]},{"osd":13,"uuid":"89c43816-fffc-42a6-98af-7f215631d595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6809","nonce":84283894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6811","nonce":84283894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6815","nonce":84283894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":84283894},{"type":"v1","addr":"172.21.15.118:6813","nonce":84283894}]},"public_addr":"172.21.15.118:6809/84283894","cluster_addr":"172.21.15.118:6811/84283894","heartbeat_back_addr":"172.21.15.118:6815/84283894","heartbeat_front_addr":"172.21.15.118:6813/84283894","state":["exists","up"]},{"osd":14,"uuid":"33cb3383-2d68-4928-be43-cfc215f6b080","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6801","nonce":1752008388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6803","nonce":1752008388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6807","nonce":1752008388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":1752008388},{"type":"v1","addr":"172.21.15.177:6805","nonce":1752008388}]},"public_addr":"172.21.15.177:6801/1752008388","cluster_addr":"172.21.15.177:6803/1752008388","heartbeat_back_addr":"172.21.15.177:6807/1752008388","heartbeat_front_addr":"172.21.15.177:6805/1752008388","state":["exists","up"]},{"osd":15,"uuid":"b6c6a0e1-a5bd-4be7-a77b-b5b6f3ea9a4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6803","nonce":3564981919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6805","nonce":3564981919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6808","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6809","nonce":3564981919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":3564981919},{"type":"v1","addr":"172.21.15.195:6807","nonce":3564981919}]},"public_addr":"172.21.15.195:6803/3564981919","cluster_addr":"172.21.15.195:6805/3564981919","heartbeat_back_addr":"172.21.15.195:6809/3564981919","heartbeat_front_addr":"172.21.15.195:6807/3564981919","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:10.406807+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:20.891965+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:25.986137+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:30.530796+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:10.699837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:09.304914+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:20.841771+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:22.263679+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:18.765693+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:26.923353+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:28.101178+0000","dead_epoch":82},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.748260+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:19.298920+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.709317+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:15:23.987870+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T02:16:21.522681+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"},"shecprofile":{"c":"2","crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","k":"4","m":"3","name":"shecprofile","plugin":"shec","technique":"multiple"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T02:16:35.165 INFO:tasks.ceph:Scrubbing osd.0 2024-09-16T02:16:35.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:35.345 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:35.345 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:35.345 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:35.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-16T02:16:35.702 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T02:16:35.722 INFO:tasks.ceph:Scrubbing osd.4 2024-09-16T02:16:35.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:35.890 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:35.890 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:35.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-16T02:16:36.214 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 4 to deep-scrub 2024-09-16T02:16:36.231 INFO:tasks.ceph:Scrubbing osd.8 2024-09-16T02:16:36.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:36.393 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:36.393 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:36.394 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:36.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-16T02:16:36.721 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 8 to deep-scrub 2024-09-16T02:16:36.739 INFO:tasks.ceph:Scrubbing osd.12 2024-09-16T02:16:36.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:36.903 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:36.903 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:36.903 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:36.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-16T02:16:37.242 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 12 to deep-scrub 2024-09-16T02:16:37.261 INFO:tasks.ceph:Scrubbing osd.1 2024-09-16T02:16:37.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:37.434 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:37.434 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:37.434 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:37.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-16T02:16:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T02:16:37.784 INFO:tasks.ceph:Scrubbing osd.5 2024-09-16T02:16:37.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:37.946 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:37.947 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:37.947 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:37.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-16T02:16:38.278 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 5 to deep-scrub 2024-09-16T02:16:38.298 INFO:tasks.ceph:Scrubbing osd.9 2024-09-16T02:16:38.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:38.461 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:38.461 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:38.461 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:38.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-16T02:16:38.785 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 9 to deep-scrub 2024-09-16T02:16:38.803 INFO:tasks.ceph:Scrubbing osd.13 2024-09-16T02:16:38.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:38.962 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:38.962 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:38.962 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:38.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-16T02:16:39.291 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 13 to deep-scrub 2024-09-16T02:16:39.310 INFO:tasks.ceph:Scrubbing osd.2 2024-09-16T02:16:39.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:39.467 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:39.468 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:39.468 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:39.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-16T02:16:39.784 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T02:16:39.801 INFO:tasks.ceph:Scrubbing osd.6 2024-09-16T02:16:39.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:39.956 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:39.956 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:39.956 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:39.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-16T02:16:40.290 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 6 to deep-scrub 2024-09-16T02:16:40.310 INFO:tasks.ceph:Scrubbing osd.10 2024-09-16T02:16:40.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:40.480 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:40.480 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:40.480 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:40.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-16T02:16:40.815 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 10 to deep-scrub 2024-09-16T02:16:40.836 INFO:tasks.ceph:Scrubbing osd.14 2024-09-16T02:16:40.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:41.004 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:41.004 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:41.004 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:41.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-16T02:16:41.333 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 14 to deep-scrub 2024-09-16T02:16:41.351 INFO:tasks.ceph:Scrubbing osd.3 2024-09-16T02:16:41.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:41.506 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:41.506 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:41.506 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:41.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-16T02:16:41.824 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 3 to deep-scrub 2024-09-16T02:16:41.844 INFO:tasks.ceph:Scrubbing osd.7 2024-09-16T02:16:41.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:42.016 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:42.016 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:42.016 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:42.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-16T02:16:42.332 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 7 to deep-scrub 2024-09-16T02:16:42.350 INFO:tasks.ceph:Scrubbing osd.11 2024-09-16T02:16:42.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:42.517 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:42.517 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:42.517 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:42.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-16T02:16:42.851 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 11 to deep-scrub 2024-09-16T02:16:42.872 INFO:tasks.ceph:Scrubbing osd.15 2024-09-16T02:16:42.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-16T02:16:43.043 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-09-16T02:16:43.043 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T02:16:43.043 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-09-16T02:16:43.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-16T02:16:43.378 INFO:teuthology.orchestra.run.smithi003.stderr:instructed osd(s) 15 to deep-scrub 2024-09-16T02:16:43.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:16:43.715 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:16:43.715 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:16:43.733 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-16T02:16:42.039700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":528136,"kb_used_data":28516,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509421304,"statfs":{"total":1546188226560,"available":1545647415296,"internally_reserved":0,"allocated":29200384,"data_stored":23236284,"data_compressed":6331022,"data_compressed_allocated":8679424,"data_compressed_original":18219008,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002043"},"pg_stats":[{"pgid":"1.0","version":"26'112","reported_seq":407,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.321101+0000","last_change":"2024-09-16T02:16:33.384065+0000","last_active":"2024-09-16T02:16:39.321101+0000","last_peered":"2024-09-16T02:16:39.321101+0000","last_clean":"2024-09-16T02:16:39.321101+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:39.321101+0000","last_undegraded":"2024-09-16T02:16:39.321101+0000","last_fullsized":"2024-09-16T02:16:39.321101+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'112","last_scrub_stamp":"2024-09-16T02:16:33.383849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:05.038727+0000","last_clean_scrub_stamp":"2024-09-16T02:16:33.383849+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:39.206940+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"153'2","reported_seq":280,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.435205+0000","last_change":"2024-09-16T02:16:39.073358+0000","last_active":"2024-09-16T02:16:39.435205+0000","last_peered":"2024-09-16T02:16:39.435205+0000","last_clean":"2024-09-16T02:16:39.435205+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:39.435205+0000","last_undegraded":"2024-09-16T02:16:39.435205+0000","last_fullsized":"2024-09-16T02:16:39.435205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'2","last_scrub_stamp":"2024-09-16T02:16:39.073193+0000","last_deep_scrub":"153'2","last_deep_scrub_stamp":"2024-09-16T02:16:39.073193+0000","last_clean_scrub_stamp":"2024-09-16T02:16:39.073193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:50.083331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":282,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:40.028610+0000","last_change":"2024-09-16T02:16:40.028610+0000","last_active":"2024-09-16T02:16:40.028610+0000","last_peered":"2024-09-16T02:16:40.028610+0000","last_clean":"2024-09-16T02:16:40.028610+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:40.028610+0000","last_undegraded":"2024-09-16T02:16:40.028610+0000","last_fullsized":"2024-09-16T02:16:40.028610+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:40.028451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:40.028451+0000","last_clean_scrub_stamp":"2024-09-16T02:16:40.028451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:55.465963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":271,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.321136+0000","last_change":"2024-09-16T02:16:20.324063+0000","last_active":"2024-09-16T02:16:39.321136+0000","last_peered":"2024-09-16T02:16:39.321136+0000","last_clean":"2024-09-16T02:16:39.321136+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:39.321136+0000","last_undegraded":"2024-09-16T02:16:39.321136+0000","last_fullsized":"2024-09-16T02:16:39.321136+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:20.323879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:20.323879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:47.551808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":276,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.134225+0000","last_change":"2024-09-16T02:15:16.145482+0000","last_active":"2024-09-16T02:16:39.134225+0000","last_peered":"2024-09-16T02:16:39.134225+0000","last_clean":"2024-09-16T02:16:39.134225+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:39.134225+0000","last_undegraded":"2024-09-16T02:16:39.134225+0000","last_fullsized":"2024-09-16T02:16:39.134225+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-16T02:15:16.145312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:16.145312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":189,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.435844+0000","last_change":"2024-09-16T02:16:31.575496+0000","last_active":"2024-09-16T02:16:39.435844+0000","last_peered":"2024-09-16T02:16:39.435844+0000","last_clean":"2024-09-16T02:16:39.435844+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:39.435844+0000","last_undegraded":"2024-09-16T02:16:39.435844+0000","last_fullsized":"2024-09-16T02:16:39.435844+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:31.575335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:16:31.575335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:52.788053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T02:16:37.639325+0000","last_change":"2024-09-16T02:16:37.639325+0000","last_active":"2024-09-16T02:16:37.639325+0000","last_peered":"2024-09-16T02:16:37.639325+0000","last_clean":"2024-09-16T02:16:37.639325+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:37.639325+0000","last_undegraded":"2024-09-16T02:16:37.639325+0000","last_fullsized":"2024-09-16T02:16:37.639325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:37.639173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:37.639173+0000","last_clean_scrub_stamp":"2024-09-16T02:16:37.639173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:48.308701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":273,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-16T02:16:37.149795+0000","last_change":"2024-09-16T02:15:26.947081+0000","last_active":"2024-09-16T02:16:37.149795+0000","last_peered":"2024-09-16T02:16:37.149795+0000","last_clean":"2024-09-16T02:16:37.149795+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:37.149795+0000","last_undegraded":"2024-09-16T02:16:37.149795+0000","last_fullsized":"2024-09-16T02:16:37.149795+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:26.946897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:26.946897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:48.196457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":275,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-09-16T02:16:39.435575+0000","last_change":"2024-09-16T02:15:28.735081+0000","last_active":"2024-09-16T02:16:39.435575+0000","last_peered":"2024-09-16T02:16:39.435575+0000","last_clean":"2024-09-16T02:16:39.435575+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:39.435575+0000","last_undegraded":"2024-09-16T02:16:39.435575+0000","last_fullsized":"2024-09-16T02:16:39.435575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:15:28.734924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:14:07.079827+0000","last_clean_scrub_stamp":"2024-09-16T02:15:28.734924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:16:58.437221+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1903643,"data_compressed":420591,"data_compressed_allocated":573440,"data_compressed_original":1576960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32972,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338868,"statfs":{"total":96636764160,"available":96603000832,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280548,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37136,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334704,"statfs":{"total":96636764160,"available":96598736896,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32948,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338892,"statfs":{"total":96636764160,"available":96603025408,"internally_reserved":0,"allocated":1802240,"data_stored":1374715,"data_compressed":388060,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280548,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32960,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1814528,"data_stored":1384365,"data_compressed":391059,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313250,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32948,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338892,"statfs":{"total":96636764160,"available":96603025408,"internally_reserved":0,"allocated":1802240,"data_stored":1374715,"data_compressed":388060,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280548,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32984,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338856,"statfs":{"total":96636764160,"available":96602988544,"internally_reserved":0,"allocated":1830912,"data_stored":1393984,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280547,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32960,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338880,"statfs":{"total":96636764160,"available":96603013120,"internally_reserved":0,"allocated":1814528,"data_stored":1384365,"data_compressed":391059,"data_compressed_allocated":536576,"data_compressed_original":1073152,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280546,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1788,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1830912,"data_stored":1393984,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32948,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338892,"statfs":{"total":96636764160,"available":96603025408,"internally_reserved":0,"allocated":1802240,"data_stored":1374715,"data_compressed":388060,"data_compressed_allocated":532480,"data_compressed_original":1064960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280547,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1888256,"data_stored":1922893,"data_compressed":426590,"data_compressed_allocated":581632,"data_compressed_original":1593344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28820,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343020,"statfs":{"total":96636764160,"available":96607252480,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280547,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280547,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32976,"kb_used_data":1784,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338864,"statfs":{"total":96636764160,"available":96602996736,"internally_reserved":0,"allocated":1826816,"data_stored":1393965,"data_compressed":394059,"data_compressed_allocated":540672,"data_compressed_original":1081344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729616,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29352,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342488,"statfs":{"total":96636764160,"available":96606707712,"internally_reserved":0,"allocated":1789952,"data_stored":1365115,"data_compressed":385071,"data_compressed_allocated":528384,"data_compressed_original":1056768,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:16:43.737 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T02:16:33.383849+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=33, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.737 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T02:16:20.323879+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=20, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.737 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T02:15:16.145312+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=15, tm_sec=16, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.737 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T02:16:31.575335+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=31, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.737 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T02:15:26.946897+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=15, tm_sec=26, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.738 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T02:15:28.734924+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=15, tm_sec=28, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=2, tm_min=16, tm_sec=34, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T02:16:43.738 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T02:17:03.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-16T02:17:04.069 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-09-16T02:17:04.069 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-09-16T02:17:04.087 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-16T02:17:04.047331+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":9,"kb":1509949440,"kb_used":528880,"kb_used_data":29248,"kb_used_omap":24,"kb_used_meta":499239,"kb_avail":1509420560,"statfs":{"total":1546188226560,"available":1545646653440,"internally_reserved":0,"allocated":29949952,"data_stored":23822934,"data_compressed":6514086,"data_compressed_allocated":8929280,"data_compressed_original":18718720,"omap_allocated":25170,"internal_metadata":511221166},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004102"},"pg_stats":[{"pgid":"1.0","version":"154'114","reported_seq":423,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:50.257601+0000","last_change":"2024-09-16T02:16:41.473269+0000","last_active":"2024-09-16T02:16:50.257601+0000","last_peered":"2024-09-16T02:16:50.257601+0000","last_clean":"2024-09-16T02:16:50.257601+0000","last_became_active":"2024-09-16T02:14:06.358056+0000","last_became_peered":"2024-09-16T02:14:06.358056+0000","last_unstale":"2024-09-16T02:16:50.257601+0000","last_undegraded":"2024-09-16T02:16:50.257601+0000","last_fullsized":"2024-09-16T02:16:50.257601+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'114","last_scrub_stamp":"2024-09-16T02:16:41.473068+0000","last_deep_scrub":"154'114","last_deep_scrub_stamp":"2024-09-16T02:16:41.473068+0000","last_clean_scrub_stamp":"2024-09-16T02:16:41.473068+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:18:10.269165+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,0],"acting":[13,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.3","version":"153'2","reported_seq":286,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.189117+0000","last_change":"2024-09-16T02:16:39.073358+0000","last_active":"2024-09-16T02:16:49.189117+0000","last_peered":"2024-09-16T02:16:49.189117+0000","last_clean":"2024-09-16T02:16:49.189117+0000","last_became_active":"2024-09-16T02:14:08.171463+0000","last_became_peered":"2024-09-16T02:14:08.171463+0000","last_unstale":"2024-09-16T02:16:49.189117+0000","last_undegraded":"2024-09-16T02:16:49.189117+0000","last_fullsized":"2024-09-16T02:16:49.189117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'2","last_scrub_stamp":"2024-09-16T02:16:39.073193+0000","last_deep_scrub":"153'2","last_deep_scrub_stamp":"2024-09-16T02:16:39.073193+0000","last_clean_scrub_stamp":"2024-09-16T02:16:39.073193+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:50.083331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00099030400000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":287,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.189567+0000","last_change":"2024-09-16T02:16:40.028610+0000","last_active":"2024-09-16T02:16:49.189567+0000","last_peered":"2024-09-16T02:16:49.189567+0000","last_clean":"2024-09-16T02:16:49.189567+0000","last_became_active":"2024-09-16T02:14:08.174295+0000","last_became_peered":"2024-09-16T02:14:08.174295+0000","last_unstale":"2024-09-16T02:16:49.189567+0000","last_undegraded":"2024-09-16T02:16:49.189567+0000","last_fullsized":"2024-09-16T02:16:49.189567+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:40.028451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:40.028451+0000","last_clean_scrub_stamp":"2024-09-16T02:16:40.028451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:55.465963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081492800000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":285,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:50.257660+0000","last_change":"2024-09-16T02:16:40.314830+0000","last_active":"2024-09-16T02:16:50.257660+0000","last_peered":"2024-09-16T02:16:50.257660+0000","last_clean":"2024-09-16T02:16:50.257660+0000","last_became_active":"2024-09-16T02:14:08.173032+0000","last_became_peered":"2024-09-16T02:14:08.173032+0000","last_unstale":"2024-09-16T02:16:50.257660+0000","last_undegraded":"2024-09-16T02:16:50.257660+0000","last_fullsized":"2024-09-16T02:16:50.257660+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:40.314669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:40.314669+0000","last_clean_scrub_stamp":"2024-09-16T02:16:40.314669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:50.703991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095138999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[13,5],"acting":[13,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":13,"acting_primary":13,"purged_snaps":[]},{"pgid":"2.2","version":"154'3","reported_seq":290,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.189556+0000","last_change":"2024-09-16T02:16:40.982584+0000","last_active":"2024-09-16T02:16:49.189556+0000","last_peered":"2024-09-16T02:16:49.189556+0000","last_clean":"2024-09-16T02:16:49.189556+0000","last_became_active":"2024-09-16T02:14:08.855848+0000","last_became_peered":"2024-09-16T02:14:08.855848+0000","last_unstale":"2024-09-16T02:16:49.189556+0000","last_undegraded":"2024-09-16T02:16:49.189556+0000","last_fullsized":"2024-09-16T02:16:49.189556+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'3","last_scrub_stamp":"2024-09-16T02:16:40.982427+0000","last_deep_scrub":"154'3","last_deep_scrub_stamp":"2024-09-16T02:16:40.982427+0000","last_clean_scrub_stamp":"2024-09-16T02:16:40.982427+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:48.460116+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002004198,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":204,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.626327+0000","last_change":"2024-09-16T02:16:43.563733+0000","last_active":"2024-09-16T02:16:49.626327+0000","last_peered":"2024-09-16T02:16:49.626327+0000","last_clean":"2024-09-16T02:16:49.626327+0000","last_became_active":"2024-09-16T02:14:55.927035+0000","last_became_peered":"2024-09-16T02:14:55.927035+0000","last_unstale":"2024-09-16T02:16:49.626327+0000","last_undegraded":"2024-09-16T02:16:49.626327+0000","last_fullsized":"2024-09-16T02:16:49.626327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:43.563516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:43.563516+0000","last_clean_scrub_stamp":"2024-09-16T02:16:43.563516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:55.783750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,15],"acting":[11,15],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.442562+0000","last_change":"2024-09-16T02:16:37.639325+0000","last_active":"2024-09-16T02:16:49.442562+0000","last_peered":"2024-09-16T02:16:49.442562+0000","last_clean":"2024-09-16T02:16:49.442562+0000","last_became_active":"2024-09-16T02:14:08.853852+0000","last_became_peered":"2024-09-16T02:14:08.853852+0000","last_unstale":"2024-09-16T02:16:49.442562+0000","last_undegraded":"2024-09-16T02:16:49.442562+0000","last_fullsized":"2024-09-16T02:16:49.442562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:37.639173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:37.639173+0000","last_clean_scrub_stamp":"2024-09-16T02:16:37.639173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:48.308701+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058508300000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7],"acting":[8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":291,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.441905+0000","last_change":"2024-09-16T02:16:43.989662+0000","last_active":"2024-09-16T02:16:49.441905+0000","last_peered":"2024-09-16T02:16:49.441905+0000","last_clean":"2024-09-16T02:16:49.441905+0000","last_became_active":"2024-09-16T02:14:08.172510+0000","last_became_peered":"2024-09-16T02:14:08.172510+0000","last_unstale":"2024-09-16T02:16:49.441905+0000","last_undegraded":"2024-09-16T02:16:49.441905+0000","last_fullsized":"2024-09-16T02:16:49.441905+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:43.989507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:43.989507+0000","last_clean_scrub_stamp":"2024-09-16T02:16:43.989507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:18:08.661967+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00083969700000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,0],"acting":[15,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":157,"state":"active+clean","last_fresh":"2024-09-16T02:16:49.441221+0000","last_change":"2024-09-16T02:16:40.484668+0000","last_active":"2024-09-16T02:16:49.441221+0000","last_peered":"2024-09-16T02:16:49.441221+0000","last_clean":"2024-09-16T02:16:49.441221+0000","last_became_active":"2024-09-16T02:14:08.171745+0000","last_became_peered":"2024-09-16T02:14:08.171745+0000","last_unstale":"2024-09-16T02:16:49.441221+0000","last_undegraded":"2024-09-16T02:16:49.441221+0000","last_fullsized":"2024-09-16T02:16:49.441221+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T02:16:40.484471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T02:16:40.484471+0000","last_clean_scrub_stamp":"2024-09-16T02:16:40.484471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T02:17:49.753560+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001113689,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,13],"acting":[6,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":10},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65062,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":13,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1925120,"data_stored":1951743,"data_compressed":435595,"data_compressed_allocated":593920,"data_compressed_original":1617920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37168,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334672,"statfs":{"total":96636764160,"available":96598704128,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,3,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":21,"seq":90194313256,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,7,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":22,"seq":94489280552,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":15,"up_from":21,"seq":90194313254,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,6,8,9,10,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280552,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1867776,"data_stored":1422834,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,6,7,9,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280551,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280550,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33016,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338824,"statfs":{"total":96636764160,"available":96602955776,"internally_reserved":0,"allocated":1867776,"data_stored":1422834,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,6,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280551,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,6,7,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280551,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33072,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338768,"statfs":{"total":96636764160,"available":96602898432,"internally_reserved":0,"allocated":1925120,"data_stored":1951743,"data_compressed":435595,"data_compressed_allocated":593920,"data_compressed_original":1617920,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28852,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342988,"statfs":{"total":96636764160,"available":96607219712,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":22,"seq":94489280552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33008,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338832,"statfs":{"total":96636764160,"available":96602963968,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[3,4,5,7,8,9,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280551,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":33012,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338828,"statfs":{"total":96636764160,"available":96602959872,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,8,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":100,"seq":429496729621,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":29428,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":27582,"kb_avail":94342412,"statfs":{"total":96636764160,"available":96606629888,"internally_reserved":0,"allocated":1863680,"data_stored":1422815,"data_compressed":403064,"data_compressed_allocated":552960,"data_compressed_original":1105920,"omap_allocated":1248,"internal_metadata":28244768},"hb_peers":[0,3,4,7,8,9,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32531,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T02:17:04.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-16T02:17:04.429 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T02:17:04.429 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T02:17:04.430 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-16T02:17:04.430 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:04.492 INFO:tasks.ceph.osd.0:Stopped 2024-09-16T02:17:04.492 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-16T02:17:04.493 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:04.586 INFO:tasks.ceph.osd.4:Stopped 2024-09-16T02:17:04.586 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-16T02:17:04.586 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:04.675 INFO:tasks.ceph.osd.8:Stopped 2024-09-16T02:17:04.675 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-16T02:17:04.675 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:10.777 INFO:tasks.ceph.osd.12:Stopped 2024-09-16T02:17:10.778 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-16T02:17:10.778 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:10.843 INFO:tasks.ceph.osd.1:Stopped 2024-09-16T02:17:10.843 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-16T02:17:10.843 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:10.916 INFO:tasks.ceph.osd.5:Stopped 2024-09-16T02:17:10.916 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-16T02:17:10.916 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:11.016 INFO:tasks.ceph.osd.9:Stopped 2024-09-16T02:17:11.016 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-16T02:17:11.017 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:11.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 is failed for ~0s 2024-09-16T02:17:17.118 INFO:tasks.ceph.osd.13:Stopped 2024-09-16T02:17:17.118 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-16T02:17:17.118 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:17.196 INFO:tasks.ceph.osd.2:Stopped 2024-09-16T02:17:17.197 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-16T02:17:17.197 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:17.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 is failed for ~0s 2024-09-16T02:17:17.702 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.12 has been restored 2024-09-16T02:17:23.297 INFO:tasks.ceph.osd.6:Stopped 2024-09-16T02:17:23.298 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-16T02:17:23.298 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.349 INFO:tasks.ceph.osd.10:Stopped 2024-09-16T02:17:23.349 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-16T02:17:23.350 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.440 INFO:tasks.ceph.osd.14:Stopped 2024-09-16T02:17:23.440 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-16T02:17:23.440 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.503 INFO:tasks.ceph.osd.3:Stopped 2024-09-16T02:17:23.503 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-16T02:17:23.503 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.595 INFO:tasks.ceph.osd.7:Stopped 2024-09-16T02:17:23.596 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-16T02:17:23.596 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.651 INFO:tasks.ceph.osd.11:Stopped 2024-09-16T02:17:23.652 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-16T02:17:23.652 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.714 INFO:tasks.ceph.osd.15:Stopped 2024-09-16T02:17:23.714 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T02:17:23.714 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-16T02:17:23.714 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.742 INFO:tasks.ceph.mgr.y:Stopped 2024-09-16T02:17:23.743 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-16T02:17:23.743 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-09-16T02:17:23.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-09-16T02:17:23.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.13 has been restored 2024-09-16T02:17:23.776 INFO:tasks.ceph.mgr.x:Stopped 2024-09-16T02:17:23.776 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T02:17:23.776 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-16T02:17:23.776 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.804 INFO:tasks.ceph.mon.a:Stopped 2024-09-16T02:17:23.804 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-16T02:17:23.804 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.830 INFO:tasks.ceph.mon.b:Stopped 2024-09-16T02:17:23.830 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-16T02:17:23.830 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T02:17:23.855 INFO:tasks.ceph.mon.c:Stopped 2024-09-16T02:17:23.855 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T02:17:23.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'slow request' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-16T02:17:23.875 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:17:23.875 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-16T02:17:24.192 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:17:24.193 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-16T02:17:24.426 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:17:24.426 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-16T02:17:24.602 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:17:24.602 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-16T02:17:24.793 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:17:24.794 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-16T02:17:24.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:17:24.996 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-16T02:17:25.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:17:25.222 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-16T02:17:25.447 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:17:25.448 DEBUG:teuthology.orchestra.run.smithi118:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-16T02:17:25.665 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:17:25.666 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-16T02:17:25.907 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:17:25.907 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-16T02:17:26.059 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:17:26.059 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-16T02:17:26.184 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:17:26.184 DEBUG:teuthology.orchestra.run.smithi177:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-16T02:17:26.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:17:26.334 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-16T02:17:26.535 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:17:26.535 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-16T02:17:26.760 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:17:26.760 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-16T02:17:26.941 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:17:26.942 DEBUG:teuthology.orchestra.run.smithi195:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-16T02:17:27.184 INFO:tasks.ceph:Archiving mon data... 2024-09-16T02:17:27.185 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/data/mon.a.tgz 2024-09-16T02:17:27.186 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-09-16T02:17:27.191 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.nsTr12YiKz 2024-09-16T02:17:27.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.nsTr12YiKz 2024-09-16T02:17:27.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.nsTr12YiKz 2024-09-16T02:17:27.417 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.nsTr12YiKz is 470KB 2024-09-16T02:17:27.480 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.nsTr12YiKz 2024-09-16T02:17:27.485 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/data/mon.b.tgz 2024-09-16T02:17:27.485 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-09-16T02:17:27.491 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.ikjm488j4T 2024-09-16T02:17:27.492 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ikjm488j4T 2024-09-16T02:17:27.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.ikjm488j4T 2024-09-16T02:17:27.746 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.ikjm488j4T is 649KB 2024-09-16T02:17:27.809 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.ikjm488j4T 2024-09-16T02:17:27.814 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/data/mon.c.tgz 2024-09-16T02:17:27.814 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-09-16T02:17:27.820 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.SQxFMSKGeH 2024-09-16T02:17:27.820 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.SQxFMSKGeH 2024-09-16T02:17:28.017 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.SQxFMSKGeH 2024-09-16T02:17:28.077 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.SQxFMSKGeH is 649KB 2024-09-16T02:17:28.139 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.SQxFMSKGeH 2024-09-16T02:17:28.144 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T02:17:28.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T02:17:28.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T02:17:28.149 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T02:17:28.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T02:17:28.207 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T02:17:28.212 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T02:17:28.250 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T02:17:28.255 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-16T02:17:28.255 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T02:17:28.257 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi003/crash 2024-09-16T02:17:28.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T02:17:28.271 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi118/crash 2024-09-16T02:17:28.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T02:17:28.285 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi177/crash 2024-09-16T02:17:28.286 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T02:17:28.305 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi195/crash 2024-09-16T02:17:28.306 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T02:17:28.319 INFO:tasks.ceph:Compressing logs... 2024-09-16T02:17:28.319 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T02:17:28.322 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T02:17:28.332 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T02:17:28.342 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-09-16T02:17:28.343 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36674.log 2024-09-16T02:17:28.343 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-09-16T02:17:28.343 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-09-16T02:17:28.344 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36674.log.gz 2024-09-16T02:17:28.344 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29753.log 2024-09-16T02:17:28.345 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29912.log 2024-09-16T02:17:28.345 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37102.log: /var/log/ceph/ceph-client.admin.29753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29753.log.gz 2024-09-16T02:17:28.345 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz -5 2024-09-16T02:17:28.345 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30806.log 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33900.log 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29912.log.gz 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30806.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29902.log 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30806.log.gz 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33900.log.gz 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37056.log 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29902.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29902.log.gz 2024-09-16T02:17:28.347 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37056.log.gz 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32653.log 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30282.log 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32653.log: 24.6% -- replaced with /var/log/ceph/ceph-client.admin.32653.log.gz 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36061.log 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33596.log 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30282.log.gz 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36061.log.gz 2024-09-16T02:17:28.349 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32849.log 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.33596.log.gz 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35749.log 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30379.log 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32849.log.gz 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37621.log 2024-09-16T02:17:28.353 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35749.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35612.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30379.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37621.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37621.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37465.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35612.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37725.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36035.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37465.log.gz 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36581.log 2024-09-16T02:17:28.354 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37725.log.gz 2024-09-16T02:17:28.355 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T02:17:28.361 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-client.admin.36035.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36035.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36581.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36689.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32366.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36689.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31818.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34245.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32366.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33827.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31818.log.gz 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36555.log 2024-09-16T02:17:28.362 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34245.log.gz 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38427.log/var/log/ceph/ceph-client.admin.33827.log: 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.33827.log.gz 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36555.log.gz 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31330.log 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33461.log 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38427.log.gz 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31330.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.31330.log.gz -- 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.37595.log 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29709.log 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr: 24.0% -- replaced with /var/log/ceph/ceph-client.admin.33461.log.gz 2024-09-16T02:17:28.363 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34061.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37595.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29709.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29709.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.33002.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35479.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34061.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33002.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30333.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38193.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35479.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32815.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30333.log.gz 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-client.admin.38193.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36693.log 2024-09-16T02:17:28.364 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38193.log.gz 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31209.log 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32815.log.gz 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36693.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36693.log.gz 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36321.log 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31926.log 2024-09-16T02:17:28.365 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31209.log.gz 2024-09-16T02:17:28.376 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32038.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30136.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32025.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/bluestore_tool.32038.log: /var/log/ceph/objectstore_tool.30136.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/bluestore_tool.32025.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30332.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr: 86.2% -- replaced with /var/log/ceph/bluestore_tool.32025.log.gz 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/objectstore_tool.30332.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30528.log 2024-09-16T02:17:28.377 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-mon.c.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/objectstore_tool.30528.log: 94.2% -- replaced with /var/log/ceph/objectstore_tool.30136.log.gz 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32439.log 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph.audit.log: 93.8% -- replaced with /var/log/ceph/bluestore_tool.32038.log.gz 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/bluestore_tool.32812.log 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr: 94.0% -- replaced with /var/log/ceph/objectstore_tool.30332.log.gz 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/bluestore_tool.32439.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T02:17:28.384 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/bluestore_tool.32812.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31272.log 2024-09-16T02:17:28.393 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/ceph-osd.10.log: 92.1% -- replaced with /var/log/ceph/bluestore_tool.32812.log.gz 2024-09-16T02:17:28.400 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/objectstore_tool.31272.log: 93.6% -- replaced with /var/log/ceph/bluestore_tool.32439.log.gz 2024-09-16T02:17:28.407 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-16T02:17:28.407 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-16T02:17:28.407 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T02:17:28.407 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T02:17:28.407 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-16T02:17:28.408 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-16T02:17:28.408 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-16T02:17:28.408 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T02:17:28.408 INFO:teuthology.orchestra.run.smithi118.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T02:17:28.410 INFO:teuthology.orchestra.run.smithi177.stderr: 93.5% -- replaced with /var/log/ceph/objectstore_tool.31272.log.gz 2024-09-16T02:17:28.411 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-09-16T02:17:28.411 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26254.log 2024-09-16T02:17:28.411 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26278.log 2024-09-16T02:17:28.411 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26254.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26502.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30025.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26278.log: /var/log/ceph/ceph-client.admin.26502.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26278.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26502.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26478.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.0.30025.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr: 74.8% -- replaced with /var/log/ceph/ceph-client.0.30025.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26453.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26478.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26230.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26453.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26453.log.gz 2024-09-16T02:17:28.412 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26230.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.26230.log.gz 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr: /var/log/ceph/ceph-client.admin.26526.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26526.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26526.log.gz 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-osd.15.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26550.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26326.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26182.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26550.log.gz 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26381.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26326.log.gz 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26182.log.gz 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26405.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26206.log 2024-09-16T02:17:28.413 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26381.log.gz 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26302.log 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26405.log.gz 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26206.log.gz 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26429.log 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:gzip/var/log/ceph/ceph-client.admin.26302.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.26350.log 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26302.log.gz 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26429.log.gz 2024-09-16T02:17:28.414 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/ceph-client.admin.26350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26350.log.gz 2024-09-16T02:17:28.422 INFO:teuthology.orchestra.run.smithi177.stderr: 94.2% -- replaced with /var/log/ceph/objectstore_tool.30528.log.gz 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36321.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36321.log.gz 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37082.log 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33926.log 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31926.log.gz 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37082.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.37082.log.gz 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35560.log 2024-09-16T02:17:28.461 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.33926.log: /var/log/ceph/ceph-client.admin.36347.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.33926.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35560.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36347.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36347.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34955.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-osd.8.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30318.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34955.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32027.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34493.log 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-09-16T02:17:28.462 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32027.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30749.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33435.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34493.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30749.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38560.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36686.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33435.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38560.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38560.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35156.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37058.log 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36686.log: /var/log/ceph/ceph-client.admin.35156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36686.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr:gzip 24.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.35156.log.gz 2024-09-16T02:17:28.463 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36668.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29917.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37058.log.gz 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32029.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36668.log.gz 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35931.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29917.log.gz 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32029.log.gz 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33086.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35931.log.gz 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36451.log 2024-09-16T02:17:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33086.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32736.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29901.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36451.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36687.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.32736.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29901.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29901.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31559.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31884.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36687.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31559.log.gz 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36009.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36671.log 2024-09-16T02:17:28.465 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31884.log: /var/log/ceph/ceph-client.admin.36009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31884.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36009.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29805.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26450.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36671.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30251.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29805.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26450.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33378.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-client.admin.30251.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32472.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30251.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33378.log: 23.4% -- replaced with /var/log/ceph/ceph-client.admin.33378.log.gz 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29831.log 2024-09-16T02:17:28.466 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26338.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32472.log: /var/log/ceph/ceph-client.admin.29831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32472.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29831.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30872.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36607.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26338.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30872.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37803.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36399.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36607.log: /var/log/ceph/ceph-client.admin.37803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36607.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37803.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21432.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35318.log 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36399.log.gz 2024-09-16T02:17:28.467 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.21432.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21432.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37569.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29905.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35318.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32119.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37569.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29905.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37777.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.32119.log: -- /var/log/ceph/ceph-client.admin.30696.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32119.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37777.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30180.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36654.log 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30696.log.gz 2024-09-16T02:17:28.468 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30180.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37045.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36654.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33219.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.37045.log: -- /var/log/ceph/ceph-client.admin.35853.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37045.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33219.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30830.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35957.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35853.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30830.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36690.log 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35957.log.gz 2024-09-16T02:17:28.469 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36659.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37517.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36690.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36659.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34807.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37517.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33769.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30839.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34807.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29881.log 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33769.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30839.log.gz 2024-09-16T02:17:28.470 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34140.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29881.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31235.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph-client.admin.34140.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38297.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34140.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33029.log 0.0% 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31235.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38297.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29779.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32945.log 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33029.log.gz 2024-09-16T02:17:28.471 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29779.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.32945.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30689.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29914.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35640.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30689.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38089.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29914.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35640.log.gz 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37092.log 2024-09-16T02:17:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.38089.log: /var/log/ceph/ceph-client.admin.36682.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38089.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37092.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36503.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37543.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36682.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32370.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36503.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37543.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37543.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38323.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32548.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32370.log.gz 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31485.log 2024-09-16T02:17:28.473 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38323.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32548.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30697.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35391.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31485.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30697.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33982.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37751.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35391.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33982.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26219.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37751.log.gz 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30768.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.26219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35983.log 2024-09-16T02:17:28.474 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26219.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30768.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30328.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33138.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35983.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35983.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31748.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr: 28.8% -- replaced with /var/log/ceph/ceph-client.admin.30328.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37985.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.33138.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34177.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31748.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37985.log.gz 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37829.log 2024-09-16T02:17:28.475 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35905.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34177.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34177.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37829.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37071.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35905.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38453.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36477.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37071.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30825.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38453.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36477.log.gz 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37128.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30329.log 2024-09-16T02:17:28.476 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30825.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35670.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37128.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30777.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30329.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35670.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34329.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31157.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30777.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34329.log: gzip 23.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34329.log.gz -- 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.31131.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34539.log 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31157.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31131.log.gz 2024-09-16T02:17:28.477 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30253.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34539.log: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.34539.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30253.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32206.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36139.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32206.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38375.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35130.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36139.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36139.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38375.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38375.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38531.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34269.log 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35130.log.gz 2024-09-16T02:17:28.478 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36243.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38531.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33326.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.34269.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31722.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36243.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38219.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33326.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31722.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30821.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34620.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38219.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30821.log.gz 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37469.log 2024-09-16T02:17:28.479 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35439.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr: 23.9% -- replaced with /var/log/ceph/ceph-client.admin.34620.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37469.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38584.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35439.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35505.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37002.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38584.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35505.log: gzip -5 23.8% -- replaced with /var/log/ceph/ceph-client.admin.35505.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36692.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37002.log.gz 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36373.log 2024-09-16T02:17:28.480 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31338.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36692.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36373.log: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.36373.log.gz -- 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.36691.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29907.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31338.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36691.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38037.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37079.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29907.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38037.log.gz 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35775.log 2024-09-16T02:17:28.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38167.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37079.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29885.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35775.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38167.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31291.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.29885.log: -- /var/log/ceph/ceph-client.admin.31261.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29885.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33660.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31291.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31261.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33685.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37673.log 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33660.log.gz 2024-09-16T02:17:28.482 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31615.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.33685.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38271.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37673.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31615.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38271.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36666.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32254.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36217.log/var/log/ceph/ceph-client.admin.38479.log: 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29913.log 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32254.log.gz 2024-09-16T02:17:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36217.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.36217.log.gz --verbose 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr: -- /var/log/ceph/ceph-client.admin.31967.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37106.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29913.log.gz 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34458.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.31967.log.gz 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37106.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.37106.log.gz /var/log/ceph/ceph-client.admin.32308.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35879.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34458.log: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34458.log.gz 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37881.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32308.log.gz 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35879.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.35879.log.gz --verbose 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr: -- /var/log/ceph/ceph-client.admin.33330.log 2024-09-16T02:17:28.484 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35104.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37881.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31590.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33330.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30807.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35104.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30301.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31590.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29908.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30807.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35239.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30301.log.gz 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32710.log 2024-09-16T02:17:28.485 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29908.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30776.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.35239.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37907.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32710.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35267.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30776.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37855.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37907.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35267.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29678.log -- replaced with /var/log/ceph/ceph-client.admin.35267.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30277.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37855.log.gz 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33742.log 2024-09-16T02:17:28.486 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29678.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30277.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31105.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32202.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33742.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31105.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31105.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31824.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32202.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38063.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33554.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31824.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38063.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38063.log.gz 2024-09-16T02:17:28.487 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33554.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.32143.log -- replaced with /var/log/ceph/ceph-client.admin.33554.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31617.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36165.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.32143.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38349.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31617.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31773.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36165.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33090.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38349.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37101.log 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31773.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-16T02:17:28.488 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30297.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33090.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37101.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37699.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30297.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30297.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36269.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38401.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37699.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36269.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36269.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32597.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35827.log 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38401.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32597.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32597.log.gz 2024-09-16T02:17:28.489 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29904.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35827.log.gz 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29904.log.gz 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32438.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36295.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32438.log.gz 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37933.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36295.log.gz 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30383.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31331.log 2024-09-16T02:17:28.490 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37933.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36655.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30383.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37087.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31331.log: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.31331.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30326.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36655.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36529.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37087.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36191.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30326.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29653.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36529.log.gz 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34693.log 2024-09-16T02:17:28.491 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36191.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36113.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29653.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32476.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34693.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38115.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36113.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37647.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.32476.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32093.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38115.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29915.log 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37647.log.gz 2024-09-16T02:17:28.492 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34009.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32093.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32740.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29915.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35801.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34009.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30294.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32740.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35213.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35801.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30769.log 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30294.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.35213.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35213.log.gz 2024-09-16T02:17:28.493 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31486.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34351.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30769.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37097.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31486.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38141.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34351.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32626.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37097.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37118.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38141.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30323.log 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32626.log.gz 2024-09-16T02:17:28.494 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37118.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37118.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31183.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30323.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38245.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31339.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31183.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32279.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38245.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31339.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34838.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36425.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 23.8% -- replaced with /var/log/ceph/ceph-client.admin.32279.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30797.log 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.34838.log.gz 2024-09-16T02:17:28.495 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36425.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32877.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.30797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34035.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30797.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.32877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34921.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32877.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29909.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34035.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29903.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34921.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.29909.log: /var/log/ceph/ceph-client.admin.29903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29909.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29903.log.gz 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-09-16T02:17:28.496 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31584.log 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.36670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37959.log 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.31584.log: 24.5% -- replaced with /var/log/ceph/ceph-client.admin.31584.log.gz/var/log/ceph/ceph-client.admin.35718.log: 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33269.log 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37959.log.gz 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34591.log 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.33269.log: 24.3% -- replaced with /var/log/ceph/ceph-client.admin.33269.log.gz 2024-09-16T02:17:28.497 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34591.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34591.log.gz 2024-09-16T02:17:28.498 INFO:teuthology.orchestra.run.smithi195.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-16T02:17:28.533 INFO:teuthology.orchestra.run.smithi177.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-16T02:17:28.546 INFO:teuthology.orchestra.run.smithi118.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-16T02:17:28.692 INFO:teuthology.orchestra.run.smithi177.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-16T02:17:28.754 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-16T02:17:28.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-09-16T02:17:28.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-09-16T02:17:28.790 INFO:teuthology.orchestra.run.smithi118.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-16T02:17:28.834 INFO:teuthology.orchestra.run.smithi003.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-16T02:17:28.836 INFO:teuthology.orchestra.run.smithi003.stderr: 91.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-16T02:17:28.839 INFO:teuthology.orchestra.run.smithi195.stderr: 91.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-16T02:17:28.843 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-16T02:17:28.879 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-16T02:17:28.886 INFO:teuthology.orchestra.run.smithi003.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-16T02:17:28.895 INFO:teuthology.orchestra.run.smithi003.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-16T02:17:28.896 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-09-16T02:17:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m0.572s 2024-09-16T02:17:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m2.387s 2024-09-16T02:17:28.896 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.155s 2024-09-16T02:17:28.923 INFO:teuthology.orchestra.run.smithi195.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-16T02:17:28.944 INFO:teuthology.orchestra.run.smithi177.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-16T02:17:29.007 INFO:teuthology.orchestra.run.smithi195.stderr: 91.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-16T02:17:29.309 INFO:teuthology.orchestra.run.smithi177.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-16T02:17:29.313 INFO:teuthology.orchestra.run.smithi195.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-16T02:17:29.314 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-09-16T02:17:29.314 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.945s 2024-09-16T02:17:29.314 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m2.550s 2024-09-16T02:17:29.314 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.110s 2024-09-16T02:17:29.329 INFO:teuthology.orchestra.run.smithi177.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-16T02:17:29.330 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-09-16T02:17:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m0.978s 2024-09-16T02:17:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m2.947s 2024-09-16T02:17:29.331 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.136s 2024-09-16T02:17:29.424 INFO:teuthology.orchestra.run.smithi118.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-16T02:17:29.426 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-09-16T02:17:29.426 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m1.091s 2024-09-16T02:17:29.426 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m2.407s 2024-09-16T02:17:29.426 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m0.110s 2024-09-16T02:17:29.426 INFO:tasks.ceph:Archiving logs... 2024-09-16T02:17:29.426 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi003/log 2024-09-16T02:17:29.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T02:17:29.919 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi118/log 2024-09-16T02:17:29.920 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T02:17:30.236 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi177/log 2024-09-16T02:17:30.237 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T02:17:30.609 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi195/log 2024-09-16T02:17:30.610 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T02:17:30.954 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T02:17:30.963 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T02:17:30.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T02:17:30.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T02:17:30.968 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T02:17:30.970 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T02:17:31.090 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T02:17:31.098 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T02:17:31.099 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:17:31.101 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:17:31.103 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:17:31.106 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T02:17:31.112 INFO:teuthology.orchestra.run.smithi003.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:17:31.112 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================== 2024-09-16T02:17:31.112 INFO:teuthology.orchestra.run.smithi003.stdout:+hv01.front.sepi 67.205.162.81 3 u 18 64 177 0.057 -0.500 0.850 2024-09-16T02:17:31.112 INFO:teuthology.orchestra.run.smithi003.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 177 0.102 -1.319 0.794 2024-09-16T02:17:31.113 INFO:teuthology.orchestra.run.smithi003.stdout:*hv03.front.sepi 74.6.168.72 3 u 16 64 177 0.094 -0.933 0.739 2024-09-16T02:17:31.113 INFO:teuthology.orchestra.run.smithi003.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================== 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout:+hv01.front.sepi 67.205.162.81 3 u 14 64 177 0.147 -0.281 0.940 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout:+hv02.front.sepi 63.231.80.2 3 u 18 64 177 0.131 -1.226 1.014 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout:*hv03.front.sepi 74.6.168.72 3 u 19 64 177 0.083 -0.827 1.032 2024-09-16T02:17:31.115 INFO:teuthology.orchestra.run.smithi118.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout:============================================================================== 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout:+hv01.front.sepi 67.205.162.81 3 u 18 64 177 0.072 -0.611 1.008 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 64 177 0.095 -1.539 0.972 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout:*hv03.front.sepi 74.6.168.72 3 u 26 64 177 0.120 -1.237 1.002 2024-09-16T02:17:31.117 INFO:teuthology.orchestra.run.smithi177.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout:============================================================================== 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout:+hv01.front.sepi 67.205.162.81 3 u 19 64 177 0.076 -0.035 0.725 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout:*hv02.front.sepi 63.231.80.2 3 u 18 64 177 0.071 -1.134 0.679 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout:+hv03.front.sepi 74.6.168.72 3 u 18 64 177 0.095 -2.088 1.529 2024-09-16T02:17:31.119 INFO:teuthology.orchestra.run.smithi195.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T02:17:31.120 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T02:17:31.128 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T02:17:31.129 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T02:17:31.138 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T02:17:31.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T02:17:31.154 INFO:teuthology.task.internal:Duration was 882.352901 seconds 2024-09-16T02:17:31.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T02:17:31.163 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T02:17:31.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T02:17:31.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T02:17:31.168 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T02:17:31.171 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T02:17:31.211 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T02:17:31.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-09-16T02:17:31.211 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T02:17:31.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-09-16T02:17:31.256 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T02:17:31.267 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-09-16T02:17:31.268 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T02:17:31.278 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-09-16T02:17:31.278 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T02:17:31.287 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T02:17:31.287 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T02:17:31.299 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T02:17:31.316 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T02:17:31.322 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T02:17:31.356 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T02:17:31.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T02:17:31.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T02:17:31.384 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T02:17:31.390 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T02:17:31.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T02:17:31.464 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T02:17:31.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T02:17:31.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T02:17:31.487 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T02:17:31.498 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T02:17:31.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T02:17:31.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.532 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-09-16T02:17:31.532 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.541 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-09-16T02:17:31.543 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.549 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-09-16T02:17:31.561 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-09-16T02:17:31.658 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:17:31.663 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:17:31.668 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:17:31.672 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T02:17:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T02:17:31.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T02:17:31.714 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T02:17:31.714 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi003 2024-09-16T02:17:31.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T02:17:31.738 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi118 2024-09-16T02:17:31.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T02:17:31.758 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi177 2024-09-16T02:17:31.758 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T02:17:31.781 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905743/remote/smithi195 2024-09-16T02:17:31.781 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T02:17:31.803 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T02:17:31.804 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T02:17:31.806 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T02:17:31.808 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T02:17:31.822 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T02:17:31.847 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T02:17:31.859 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T02:17:31.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T02:17:31.869 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T02:17:31.869 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T02:17:31.872 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T02:17:31.874 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T02:17:31.876 INFO:teuthology.orchestra.run.smithi003.stdout: 658625 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 02:17 /home/ubuntu/cephtest 2024-09-16T02:17:31.877 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T02:17:31.878 INFO:teuthology.orchestra.run.smithi118.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 02:17 /home/ubuntu/cephtest 2024-09-16T02:17:31.880 INFO:teuthology.orchestra.run.smithi177.stdout: 658463 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 02:17 /home/ubuntu/cephtest 2024-09-16T02:17:31.896 INFO:teuthology.orchestra.run.smithi195.stdout: 658626 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 02:17 /home/ubuntu/cephtest 2024-09-16T02:17:31.897 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T02:17:31.907 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T02:17:31.938 INFO:teuthology.run:Summary data: description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} duration: 882.3529009819031 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T02:17:31.938 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T02:17:32.021 INFO:teuthology.run:pass